Occupational training support system and content creation program for the occupational training support system

Abstract

To provide a work training support system capable of coping with various pieces of work.SOLUTION: An education support system for allowing an inexperienced worker to learn techniques of an experienced worker includes: a measuring device 12 for detecting physical quantity data of operation and posture of a virtual experiencing person and movement data of a sight line; a display device 16 for projecting a visual field image in a virtual space; and a processor 20 for outputting a visual field image to the display device 16, recording physical quantity data and movement data of a sight line as teacher data when a virtual experiencing person is an experienced worker, calculating a difference by comparing physical quantity data and movement data of a sight line with teacher data when the virtual experiencing person is an inexperienced person, and performing processing for obtaining a skill level corresponding to the difference on the basis of a predetermined prediction value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for efficiently educating workers engaged in work at construction sites, etc., and particularly to a work education support system that is suitable for situations where it is difficult for an instructor to provide direct instruction, and a content creation program used in this system. [Background technology]

[0002] Recently, workplaces have been facing a shortage of skilled and mid-career workers with the know-how, knowledge, and skills required for specific tasks.In addition, in a situation where infection control measures such as avoiding contact with others are being advocated, it is difficult to carry out regular OJT (On Job Training), making it difficult to accumulate and pass on work know-how and important points (such as how to do things, posture, and the work environment).

[0003] In this current situation, many workers do not have basic knowledge about on-site work, and because they are unable to observe the work site, they often do not understand the atmosphere depending on the type of work (for example, single work and collaborative work), and even if they take general work courses, there are many cases where they are unable to acquire practical skills.

[0004] In this context, technology such as that disclosed in Patent Document 1, which combines VR (Virtual Reality) technology with a simulated tool equipped with an actuator, is attracting attention, allowing users to safely experience dangerous actions (the impact of the kickback phenomenon) that occur when using handheld rotary tools.

[0005] Furthermore, Patent Documents 2 and 3 disclose techniques for easily and efficiently creating VR content as techniques for expanding the scope of application of VR technology. For example, Patent Document 2 discloses the configuration of a program that automatically creates VR content by applying predetermined VR actions to video files uploaded by users. Furthermore, Patent Document 3 discloses a VR content management system that creates a desired virtual space by placing pre-created VR object data in the virtual space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-114066 [Patent Document 2] Japanese Patent Application Publication No. 2018-190177 [Patent Document 3] Japanese Patent Application Publication No. 2019-168934 Summary of the Invention [Problem to be solved by the invention]

[0007] The immersive device disclosed in Patent Document 1 allows people to safely learn about the dangers of specific tasks. However, the device disclosed in Patent Document 1 is a specialized immersive device for specific tasks, and if people want to experience other techniques or tasks, they will need separate systems.

[0008] Furthermore, the VR content management and creation systems disclosed in Patent Documents 2 and 3 are configured to apply data under specific conditions in order to simplify the creation of VR content. Therefore, in order to create VR content that takes into account different movements, gazes, etc. for each task, it becomes necessary to define the movements and reactions of the content for each pattern, which requires work by someone with specialized knowledge.

[0009] Therefore, the present invention aims to provide a work training support system that can be adapted to various types of work, as well as a content creation program for the work training support system that can easily create VR content to be used with this system. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the work training support system of the present invention is an education support system that allows inexperienced workers engaged in field work to learn the skills of experienced workers, and is characterized by having: a measuring device that detects physical quantity data and eye movement data of the movements and posture of a virtual experiencer for a specific task; a display device that displays a field of view image in a virtual space; and a processing device that outputs the field of view image to the display device and, if the virtual experiencer is an experienced worker, records the physical quantity data and eye movement data as training data, and, if the virtual experiencer is an inexperienced worker, compares the physical quantity data and eye movement data with the training data to calculate a difference, and performs processing to determine a level of proficiency corresponding to the difference based on a predetermined reference value.

[0011] In addition, in a work training support system having the above-mentioned characteristics, the measuring device may preferably include a target or camera that detects the movements of the virtual experiencer, and a sensor that detects the line of sight and physical quantities based on the movements of the virtual experiencer.

[0012] In addition, in the work training support system having the above-mentioned features, it is desirable that the display device includes a head mounted display (HMD) worn by the virtual experiencer, and that various operation instructions are shown to the virtual experiencer via the HMD. With such features, the virtual experiencer can be immersed in the VR space, making the virtual work more realistic.

[0013] Furthermore, in the occupational training support system having the above-described characteristics, when the proximity distance between the objects in the display of the field of view image is equal to or less than a predetermined value, the appropriate joint state defined between the objects can be expressed and displayed. With such characteristics, it becomes possible to grasp small objects, which was previously difficult to express, and to express the accurate joint state between the objects.

[0014] In addition, in order to achieve the above-mentioned object, the content creation program for an occupational education support system according to the present invention comprises: a measuring device that detects physical quantity data and gaze movement data of the movements and posture of a virtual person experiencing a specific task; a display device that displays a field of view image in a virtual space; and a processing device that outputs the field of view image to the display device and, if the virtual person experiencing the task is an experienced worker, records the physical quantity data and gaze movement data as training data; if the virtual person experiencing the task is an unskilled worker, compares the physical quantity data and gaze movement data with the training data to calculate a difference, and performs processing to determine a proficiency level corresponding to the difference based on a predetermined reference value. This content creation program is applicable to an occupational education support system that enables unskilled workers engaged in on-site work to learn the skills of experienced workers, and is characterized in that it creates a video of work experience content by selecting and adding additional information about the unskilled worker recorded in the processing device to video taken during the virtual experience by the experienced worker, and then converts the video of the work experience content into a DataSheet corresponding to the program and outputs it. [Effects of the Invention]

[0015] The work training support system with the above-mentioned features makes it possible to create and manage work experience content corresponding to various work tasks. Furthermore, the content creation program with the above-mentioned features allows users to easily create work experience content even without specialized knowledge, because a DataSheet corresponding to the playback program is output simply by selecting and adding additional information along with the video. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of an occupational training support system according to an embodiment. [Figure 2] FIG. 2 is a flow diagram showing the overall flow of a work experience using the work training support system according to the embodiment. [Figure 3] FIG. 10 is a flow chart showing details of a work experience by a skilled worker. [Figure 4] FIG. 10 is a flow diagram showing a flow when editing work experience content. [Figure 5] FIG. 10 is a diagram showing the movement of the eyes of a skilled worker during slinging work. [Figure 6] FIG. 10 is a diagram showing the movement of the line of sight of an unskilled worker during slinging work. [Figure 7] FIG. 10 is a diagram illustrating a comparison of approximation models based on feature amounts of eye movement during transportation work. [Figure 8] FIG. 10 is a diagram showing a state at the start of work, among the field of view images during work experience by an unskilled worker. [Figure 9] FIG. 10 is a diagram showing a state in which a wrench is lifted up in a field of view image during a work experience by an unskilled worker. [Figure 10] FIG. 10 is a diagram showing a state in which a nut, which is a work target, is displayed in a visual field image during work experience by an unskilled worker. [Figure 11] FIG. 10 is a diagram showing a state in which an unskilled worker is gazing intently at a nut that is the work target, in a visual field image during a work experience by the unskilled worker. [Figure 12] FIG. 10 is a view showing a state in which an instruction is given to place a wrench on a nut to be worked on, from among the visual images during the work experience by the unskilled worker. [Figure 13] FIG. 10 is a view showing a state in which a wrench is placed on a nut (lower side) that is the work target, from among the visual field images during the work experience by an unskilled worker. [Figure 14] FIG. 10 is a view showing a state in which an instruction is given to place the second wrench on the nut (upper side) that is the work target, from the visual field image during the work experience by the unskilled worker. [Figure 15]FIG. 10 is a view showing a state in which the second wrench is placed on the nut (upper side) that is the work target, in the visual field image during the work experience by the unskilled worker. [Figure 16] FIG. 10 is a view showing a state in which a wrench attached to an upper nut is rotated counterclockwise, from among the visual images during the work experience by an unskilled worker. [Figure 17] FIG. 10 is a view showing a state in which the wrench is returned to the work tool palette, from among the field of view images during the work experience by the unskilled worker. [Figure 18] FIG. 10 is a diagram showing a state in which an unskilled worker presses an experience end button in a field of view video during a work experience by the unskilled worker. [Figure 19] FIG. 10 is a diagram illustrating an example of a radar chart showing the proficiency and work level of an unskilled worker based on skills and test results. [Figure 20] FIG. 10 is a diagram showing an example of a DataSheet. [Figure 21] FIG. 21 is an enlarged view of a portion of the DataSheet shown in FIG. 20. [Figure 22] FIG. 22 is a diagram for explaining what a command based on a part of the DataSheet shown in FIG. 21 is like. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the occupational training support system and the content creation program for the occupational training support system of the present invention will be described in detail with reference to the drawings. Note that the embodiments shown below are some of the preferred embodiments for carrying out the present invention, and even if some of the configuration is changed, they can be considered as part of the present invention as long as the effects of the system are achieved.

[0018] [composition] As shown in FIG. 1, an occupational training support system 10 according to this embodiment includes at least a measurement device 12, an external input device 14, a display device 16, an external output device 18, and a processing device.

[0019] The measuring device 12 is an element for acquiring data on the work skills of participants, such as skilled workers and unskilled workers, in the work training support system 10. Specifically, it may be a camera or sensor (such as a general camera, a target as a sensor for capturing the movement, or a sensor for detecting the position of the gaze) for acquiring image data of the participant's line of sight and body movements (movements), or a sensor (such as an acceleration sensor) for detecting the movement of the hands or body during work as a physical quantity.

[0020] The external input device 14 includes elements for detecting operations, sounds, etc. during work, as well as elements for creating and editing content, which will be described in detail later. Specifically, examples of such elements include a controller for operating the screen displayed on the display device 16 during work, a microphone for inputting sounds, a PC or tablet terminal, a mouse, and a keyboard as data input means, as well as a distance measuring sensor such as a TOF (Time of Flight) for inputting detected amounts of movement (for example, to detect the amount of movement of the target described above).

[0021] The display device 16 is an element for showing the participant a visual image during work, precautions to be taken during work, instruction information, etc. Specifically, examples include an HMD (Head Mounted Display) worn by the participant, and a fixed or portable display (monitor).

[0022] The external output device 18 is an element that allows the participant to experience physical vibrations and recoils during work. Its configurations vary widely; a simple example would be a vibration that notifies the participant when the angle or time during work reaches a predetermined value. In the future, sensors called trackers may be attached to the worker's limbs, physically transmitting the vibrations, recoil, impact, resistance, and other factors that occur during actual work to the worker, thereby achieving so-called full-body tracking. The external output device 18 also includes a speaker for outputting audio and other information. In contrast to the elements that make up the measuring device 12 and the external input device 14, elements that simply have opposing functions (input and output) may be treated as a single component. For example, an HMD can be equipped with various functions, such as a camera, a sensor for detecting line of sight, a field of view image display function, and audio input / output functions.

[0023] The processing device 20 is an element for creating content based on information such as task skill data acquired via the measuring device 12, and instruction data, voice data, and operation data input via the external input device 14, and outputting the content to the display device 16 and the external output device 18. Specifically, the processing device 20 may be configured to include an input unit 20a, an output unit 20b, a control device 20c, a calculation device 20d, a memory 20e, a voice processing device 20f, an image processing device 20g, and a large-capacity storage device 20h.

[0024] The input unit 20a and the output unit 20b are elements that act as an interface, that is, a conversion unit that exchanges data between the measurement device 12, the external input device 14, the display device 16, the external output device 18, etc. In the case of the occupational training support system 10 according to the embodiment, the measurement device 12 and the external input device 14 are connected to the input unit 20a, and the display device 16 and the external output device 18 are connected to the output unit 20b.

[0025] The control device 20c is an element that transmits various data input via the input unit 20a to the calculation device 20d and the image processing device 20g, and also performs processing to transmit data input from the calculation device 20d to the image processing device 20g and the output unit 20b.

[0026] The arithmetic unit 20d is an element that transmits and receives various data between the control unit 20c and the memory 20e, and also executes arithmetic processing in response to commands from various programs and the like.

[0027] The memory 20e is a temporary storage area for temporarily expanding various data when the arithmetic unit 20d exchanges various information with the audio processing unit 20f, the mass storage unit 20h, the input unit 20a, and the output unit 20b.

[0028] The audio processing device 20f is an element for editing the audio data temporarily expanded in the memory 20e.

[0029] The image processing device 20g is an element for editing image data recorded in a mass storage device and image data transmitted from the control device 20c.

[0030] The large-capacity storage device 20h is a long-term storage area for storing various data input via the input unit 20a, and enables data to be read in response to requests from the memory 20e, the image processing device 20g, the output unit 20b, etc. Note that the specific configuration may be an HDD, an SSD, or the like.

[0031] [Action: Work education support] Next, a work experience using the work training support system 10 according to the embodiment will be described with reference to Fig. 2 to Fig. 4. First, when using (experience) the work training support system 10 according to the embodiment, participants such as skilled workers and unskilled workers log in via the external input device 14 or the like (S100: Experiencer Login).

[0032] Next, each participant puts on VR equipment such as an HMD and sensors. Here, the VR equipment may be a multi-sensory device equipped with various data input / output functions such as the measuring device 12, external input device 14, display device 16, and external output device 18 (S110: Putting on VR Equipment). After putting on the VR equipment, the participant selects an experience category. Here, the experience category can be divided into "experienced worker" and "unskilled worker." However, the "unskilled worker" category cannot be used for the work experience unless at least one teacher data (work experience content) by an experienced worker has been completed. In other words, before work experience content based on the work experience of an experienced worker is created, the "unskilled worker (= beginner)" cannot be selected when selecting the experience category (S120).

[0033] When "experienced worker" is selected in S120, the user is prompted to select an item for the work experience (for creating work experience content) (S130: Selection of experience item). When an experience item is selected in S130, a function is activated to acquire data on the line of sight, posture, and movements of the skilled worker when performing work. Specifically, the control device 20c and the calculation device 20d in the processing device 20 are operated, and a voice recognition function for acquiring audio and video via the measuring device 12 and the external input device 14, and a recording function are activated (S140: Content creation function activation). After the content creation function is activated, the skilled worker performs the work, completing the acquisition of data during the work (S150: Work experience of skilled worker).

[0034] In the work experience of the skilled worker, work equipment is called, as shown in detail in Fig. 3. Work equipment can be called by voice input or by selection from items displayed on the display device 16, such as an HMD (S150a: Calling of work equipment by skilled worker). When a work equipment is called by the skilled worker, the processing device 20 loads the work equipment database stored in the mass storage device 20h into the memory 20e via the control device 20c and the calculation device 20d. Thereafter, the work equipment corresponding to the call is identified from the work equipment database, and an object of the corresponding work equipment is displayed in the field of view image projected on the display device 16, such as an HMD (S150b: Reading of equipment object).

[0035] After the work tool corresponding to the call in the field of view image is displayed, the skilled worker calls up the work tool. As with calling up work tool, the work tool can be called up by voice input or by selection from items displayed on the display device 16 such as an HMD (S150c: Calling of work tool by skilled worker). When the work tool is called up by the skilled worker, the processing device 20 identifies the work tool corresponding to the call, as with calling up work tool, and displays an object of the corresponding work tool in the field of view image displayed on the display device 16 such as an HMD (S150d: Reading of tool object).

[0036] Once the work equipment and work tools are called, the skilled worker performs the simulated work. Instructions are given to perform the simulated work after explaining the work content one by one (S150e: Perform simulated work (experience the work in order while talking)). The simulated work progresses while repeatedly checking comments, objects, and actions for each task. Specifically, the skilled worker first explains the work actions to be performed. The explanation of the work actions is given orally, and the processing device 20 collects this explanation as audio comments using the microphone of the external input device 14 and automatically registers them in the mass storage device 20h via the memory 20e (S150f: Automatic registration of action comments (Phase 01)). When the comments regarding the work actions are registered, the processing device 20 automatically recognizes objects related to the work (e.g., wrenches and nuts) and registers them as objects used in the work. Object recognition may be achieved by detecting objects displayed on the display device 16 or by recognizing the work equipment and work tools called up in S150a or S150c. (S150g: Automatic registration of object). Thereafter, information that changes as the skilled worker performs the work is acquired and registered. Specifically, the motion information of the skilled worker is detected via the measurement device 12 and the external input device 14, and is calculated as the amount of movement, rotation angle, number of motions, etc. of the object via the control device 20c and the calculation device 20d, and is recorded in the mass storage device 20h (S150h: Automatic registration of post-motion coordinates, angle, number of motions, etc.).

[0037] The operations and registration from S150e to S150h are repeated for each work operation. After that, when all operations corresponding to the work experience are completed, the stop (end) of the simulated work is declared or selected, and the work experience by the skilled worker ends (S150i: Stop).

[0038] Here, in the work training support system 10 according to this embodiment, by defining contact points and applying an adsorption action between objects when contact points come close to each other, it is possible to realize contact between tiny parts and the gripping of tiny parts in VR, which was previously difficult.

[0039] When the end of the simulated work is declared, the various pieces of information registered in S150f through S150h are associated and recorded as a single module (S150j: Save as Module). The recorded information of the skilled worker's work experience, stored as a module, is then converted into a DataSheet corresponding to a program for creating content for the unskilled worker's work experience and output (S150k: Output as DataSheet). After S150k, the skilled worker is prompted to select either "Experience Complete" to end the work experience or "Experience Another Item" to try another work experience. A DataSheet is a table containing commands for a specific program, such as the one shown in Figure 20. Figure 21 shows an excerpt from a DataSheet. In Figure 21, column A indicates the command name, column B indicates the control number, column C indicates the stage ID, column D indicates the flag number, column E indicates the definition name A, column F indicates the definition name B, column G indicates the X coordinate, column H indicates the Y coordinate, and column I indicates the Z coordinate. Of the command lines, line a instructs the reading of a 3D model of the nut, and line b instructs the reading of a 3D model of the wrench (see FIG. 22(A)). Line c is a command instructing the user to move the wrench closer to the nut (see FIG. 22(B)), and line d is a command to set the wrench on the nut and rotate the X-axis 12 degrees around the Y-axis with the center of the nut as the center of rotation (see FIG. 22(C)) (S160).

[0040] After that, if necessary, the skilled workers who have completed the work experience can be given a questionnaire about their work experience. The contents of the questionnaire can be, for example, whether there are any points to be careful about at each work site, or whether there are any points to be careful about in the actual work that cannot be expressed in VR. This is to be used to improve the work training support system 10 and to explain points to be careful about to the content editor, the details of which will be described later (S170: Conduct questionnaire).

[0041] The work experience by the skilled worker ends with the completion of the questionnaire. Once the base of the work experience content (S150k DataSheet) is saved and output during the work experience by the skilled worker, content editing by a content editor becomes possible. Content editing for creating content for the work experience by an unskilled worker will be explained below with reference to Figure 4.

[0042] [Edit work experience content] The content editor accesses the work training support system 10 via the external input device 14 or the like and logs in. Here, the content editor does not need to be someone with advanced knowledge of general system program editing. Specifically, it is sufficient for the content editor to be someone who has learned the content editing flow in the work training support system 10 according to the embodiment, general editing operations, and has basic work knowledge in the field (mid-level to experienced workers) (S400: Content Editor Login).

[0043] A content editor who accesses the work training support system 10 starts the content editing program. When the content editing program is first started, an item such as "Load Data Sheet" is displayed on the content editing screen. After selecting the "Load Data Sheet" item, the content editor selects the Data Sheet (hereinafter also referred to as the base Data Sheet) of the work content created by the skilled worker through work experience to edit the content. The Data Sheet recorded in the mass storage device 20h is read via the control device 20c and the arithmetic device 20d and is displayed on the display device 16 (S410: Selection of "Load Data Sheet").

[0044] Next, the content editing screen displays a "Play Video" item, and the content editor can select this to play the work video recorded in the DataSheet loaded in S410. The video is played by outputting the video data recorded in the mass storage device 20h via the image processing device 20g (S420: Select "Play Video"). If there is a part in the video being played that needs editing, select "Stop" to temporarily halt the video playback (S430: Select "Stop").

[0045] After stopping the video, the user selects either "Edit" or "Edit." "Edit" is a selection item for editing previously edited comments, etc. Therefore, the first operation is to select "Edit" (S440).

[0046] The editing process involves adding comments from skilled workers to the video, supplementary images, videos, and text, and setting the coordinates of objects, etc. Comments are selected by calling up a list of comments automatically registered by skilled workers during their work experience (Phase 01 comments in S150f) and selecting from this list (S450: Selecting Phase 01 Comments). Supplementary images, videos, and text can be added by adding images and videos of actual work at the site. Regarding text, it can be notes raised in a questionnaire administered to skilled workers during their work experience. Supplementary images and videos can be selected by recording pre-recorded on-site images and videos in a large-capacity storage device, reading them via the control device 20c and the computing device 20d, and displaying them on the display device 16. Regarding text, it can be recorded as text information such as questionnaire responses (S460: Adding Images, Videos, and Text). Furthermore, setting the coordinates of an object, etc. is an act of determining the position on the display device 16 of the comment, image, video, text, etc. selected and added in S450 or S460. Coordinates can be set by inputting the coordinates of the object's placement position, or by dragging and dropping the object on the display screen to position it (S470: Setting the coordinates of Phase 03).

[0047] After setting the object coordinates, the user selects whether to add other comments, etc. The options may be, for example, "End Editing" or "Continue Editing." If "Continue Editing" is selected, steps S450 to S470 are repeated (S480).

[0048] If "End Editing" is selected in S480, a choice (Yes or No) is made as to whether or not to play the video again (S490). If "No" is selected here to not play the video, "Stop" is selected in S430. If there are any corrections to be made to comments, etc. selected and added in S450 to S470, "Edit" is selected in S440 after video playback has stopped. As for the correction items, following the steps in S450 to S470, the comment selection "Phase01 Comment", comment correction, and coordinate selection (Phase03 Coordinates) are made (S440a to S440c). When S440c is completed, the process returns to S450, where images, videos, and text are added (S460), and the coordinates of the added images, etc. are set (S470). When "Yes" is selected regarding playback of the video in S490 and editing of the entire video is completed, a DataSheet (hereinafter also referred to as a content DataSheet) including comments, additional images, etc. is output (S500: Output of DataSheet).

[0049] [Creating a working model] When the content editor finishes editing the work experience content, a work model of an experienced worker is created. Here, the work model numerically represents the characteristics of the worker's line of sight and movements, and is an element that facilitates comparison with the movements of an unskilled worker during the work experience.

[0050] The first step in creating a task model is to capture the motion and gaze data of the task participant. The motion and gaze data of the task participant is data associated with the data recorded in the content DataSheet, and is recorded in the large-capacity storage device 20h of the processing device 20, and can be read into the memory 20e (S510: Capture of task participant motion and gaze data).

[0051] Next, area division processing of movements and gaze is performed according to predetermined conditions. Area division processing is a preprocessing step for, for example, when it is desired to acquire gaze information, in which objects displayed within the field of view (work equipment, work tools, workpieces, hoisting equipment, etc.) and other workers are divided into separate areas, and it is then possible to detect where the gaze is directed within each of these divided areas (S520: area division processing of movements and gaze).

[0052] After the area division process is completed, the number of times, time, and position of gazes directed within each area (if gaze information is acquired), as well as gaze movements (sequence of gaze movement), are counted and quantified for each area. Figure 5 shows the gaze shifts, gaze duration, and number of gaze movements of an experienced worker during rigging work, divided into areas, with the size of the circle indicating the length of gaze time (the larger the circle, the longer the gaze duration) (S530: quantification process within each area). After the quantification process within each area is completed, feature values ​​for each area are extracted. Here, feature values ​​refer to the distance and speed of gaze movement, dwell time, and number of dwell times for each area (S540: extraction of feature values ​​for each area).

[0053] After extracting the feature amounts, an approximate model showing the tendency of the numerical values ​​is created, as shown in Fig. 7. In the example of the approximate linear model shown in Fig. 7, the horizontal axis shows the items of the divided area, and the vertical axis shows the gaze time for each item (S550: Create approximate model). After the approximate model is created, when all editing work related to content creation is completed, select "Editing complete" to end the editing work.

[0054] [Work experience for unskilled workers] Once the work experience content for an unskilled worker has been constructed through the above-described process of acquiring data based on the work experience of an experienced worker, editing the content based on the acquired data, and creating a work model, the work experience content for an unskilled worker can be implemented. The work experience by an unskilled worker will be described below.

[0055] First, the inexperienced worker logs in to the system as an experiencer (S100: Experiencer Login) and puts on the VR equipment (S110: Put on VR Equipment), just like the experienced worker does.

[0056] After the various equipment is fitted, the operator selects the experience category. As described above, the experience category is divided into "experienced operator" and "unskilled operator," and in this example, "unskilled operator (beginner)" is selected (S120).

[0057] When the "Unskilled Worker" item is selected in the experience category, the work experience content to be experienced is explained on the HMD display screen or through audio guidance. Here, the explanation of the work experience content may be given by displaying a character specific to the system. This is because the sense of familiarity and empathy can improve the level of work mastery (S200: Explanation of Experience Content).

[0058] After the explanation of the work experience content, the system operation via the VR equipment is explained. As with the explanation of the work experience content, this explanation may be simply by visual or audio explanation, or may be an explanation with a character displayed (S210: Explanation of system operation).

[0059] After the explanation of how to operate the system is completed, the initial movements of the inexperienced worker are detected. Differences in the body shape of the experienced worker (inexperienced worker) can result in a large error in the optimum conditions for the work when compared with the training data (work model obtained through the work experience of an experienced worker). Therefore, by detecting the initial movements, the error from the optimum conditions that arises due to differences in body shape can be corrected, allowing for accurate comparison with the training data (S220: Detection of initial movements of experienced worker).

[0060] After the error caused by the detection of the initial movement has been corrected, if the inexperienced worker is ready, they can select "Start" to begin the work experience. On the other hand, if they are unsure about operating the system or would like to hear the explanation again, they can return to S210 and listen to the explanation of how to operate the system by selecting "Listen to the operation explanation again" (S230).

[0061] When the work experience begins, an explanation of basic knowledge about the work to be experienced is given first. The explanation is given visually and / or audibly via the HMD (S240: Basic Knowledge Explanation). Next, an explanation of the basic procedures about the work to be experienced is given. As with the explanation of basic knowledge, the explanation of the work procedures is also given visually and / or audibly via the HMD (S250: Basic Work Procedure Explanation).

[0062] After the basic knowledge and explanation of the work procedures are completed, the work procedures are guided and explained via the HMD. Here, the guidance and explanation of the work procedures are given for each action involved in the work (S260: Guidance and explanation for each work procedure).

[0063] After receiving guidance and explanations about the work procedures, the inexperienced worker performs the actions (work experience) in accordance with the instructions. At this time, the measurement device detects the inexperienced worker's actions, posture, and eye movements. Based on the various detected data, the processing device 20 creates a work model in the same way as the work experience performed by an experienced worker. For example, Figure 6 shows the inexperienced worker's eye movements, gaze duration, and frequency during slinging work, broken down by area. Note that the size of the circle in Figure 6 indicates the length of gaze duration (the larger the circle, the longer the gaze duration). The processing device 20 also extracts feature values ​​from the detected data and creates a work model as shown in Figure 7.

[0064] An example of a work experience (basic work) by an inexperienced worker will now be described with reference to FIGS. 8 to 18. In the following example, the basic work that the inexperienced worker experiences is loosening the upper nut of a double nut part. First, a visual field image including the work site and tools (two wrenches) as shown in FIG. 8 is displayed on a display device 16 such as an HMD worn by the inexperienced worker. In this state, the inexperienced worker is given instructions such as "Select and lift the H20 wrench" by text display or audio guidance.

[0065] The unskilled worker who has received the instruction reaches out for the wrench in the visual field image and picks it up. At this time, the program of the work training support system 10 according to the embodiment is configured so that when the unskilled worker's hand approaches the wrench, the wrench is automatically set in the worker's hand (see FIG. 9).

[0066] Next, the unskilled worker is shown the nut to be worked on (see FIG. 10), and is instructed by text, voice, or the like to "visually check the nut for at least three seconds." The unskilled worker follows the instruction and visually inspects the nut to be worked on (see FIG. 11). After the gaze period has elapsed, the unskilled worker is instructed to "attach the H20 wrench to the nut (bottom side)" in the displayed image of FIG. 12. The unskilled worker follows the instruction and brings the H20 wrench close to the nut to be worked on. In the program of the work training support system 10 according to the embodiment, when the distance between the objects (for example, the wrench and the nut) is equal to or less than a predetermined proximity distance, the display is expanded to automatically achieve the correct assembly state (see FIG. 13).

[0067] Next, the unskilled worker is instructed to attach the lifting wrench to the nut (upper side) for the other wrench (H26 wrench), and the unskilled worker follows the instructions and attaches the H26 wrench to the upper nut (see Figures 14 and 15).The unskilled worker is then instructed to "fix the H20 wrench and turn the H26 wrench counterclockwise."The unskilled worker follows the instructions and turns only the H26 wrench counterclockwise a specified angle (for example, 30 degrees) (see Figure 16).

[0068] In the program of the work training support system 10 according to the embodiment, the display is expanded so that the connection between the work tool and the work object is automatically released when the work is completed (for example, when the wrench has rotated 30 degrees). The unskilled worker is then instructed to "return the wrench to the work tool palette," and the unskilled worker performs the movement of returning the wrench to the work tool palette in accordance with the instruction (see FIG. 17). After the series of work experiences are completed, the unskilled worker ends the work experience by pressing the experience end button (see FIG. 18) displayed on the display screen (S270: basic work experience for each work procedure).

[0069] When the basic work experience is completed, a selection screen is displayed to allow the inexperienced worker to "end" or "retry" the work experience. If the inexperienced worker selects "retry," the process returns to S260, where guidance and explanations for each work procedure are resumed. On the other hand, if the inexperienced worker selects "end," the work experience involving movements ends (S280).

[0070] When the inexperienced worker finishes the work experience, a proficiency assessment is performed based on the inexperienced worker's movements. The proficiency assessment is performed by comparing the work model of the skilled worker included in the work experience content with the work model of the inexperienced worker (see Figure 7 for an example).

[0071] The task model is compared with an approximation model (a numerical line in this embodiment), and a level is assigned depending on the degree of deviation of the angle for each parameter. For example, if the deviation from the approximation model of the skilled worker, which is the training data, is 5 degrees or less (θ≦5 degrees), the task level is determined as "Level 1," if the deviation is more than 5 degrees but less than 15 degrees (5 degrees<θ≦15 degrees), the task level is determined as "Level 2," if the deviation is more than 15 degrees but less than 30 degrees, the task level is determined as "Level 3," and if the deviation is more than 30 degrees, i.e., if the deviation is less than Level 3, a determination result is output encouraging re-learning (S290: Determination of Experienced Worker's Proficiency Based on Actions).

[0072] After the proficiency assessment based on the movements is completed, the inexperienced worker undergoes a test based on their knowledge of the on-site work and points to note about the work they have experienced. The inexperienced worker's work level is certified based on the test results (S300: Test taking and work level certification).

[0073] When the determination of the proficiency level of the unskilled worker and the certification of the work level are completed, the work training support system 10 displays the work level of the worker resulting from the work experience in a visual form. Here, the visual form includes, for example, a graph as shown in Fig. 19, as well as a display simply in numbers or scores, and a display using different colors (S310: Visualization of the experience worker's work level).

[0074] [effect] The above-described work training support system makes it possible to create work experience content corresponding to various types of work. Furthermore, editing work experience content (VR content) in the program related to this system does not require extensive programming knowledge, so even non-experts can edit the content, making it relatively easy to create various types of work experience content. [Explanation of symbols]

[0075] 10...Work training support system, 12...Measuring device, 14...External input device, 16...Display device, 18...External output device, 20...Processing device, 20a...Input section, 20b...Output section, 20c...Control device, 20d...Calculation device, 20e...Memory, 20f...Sound processing device, 20g...Image processing device, 20h...Large capacity storage device.

Claims

1. An educational support system that allows unskilled workers engaged in on-site work to learn the skills of skilled workers, a measuring device for detecting physical quantity data of the movements and postures of the virtual experiencer performing a specific task, and data of the movement of the line of sight; a display device that displays a field of view image in a virtual space; a processing device that outputs the visual field image to the display device, and records the physical quantity data and gaze movement data as training data if the virtual experiencer is an experienced worker, and compares the physical quantity data and gaze movement data with the training data if the virtual experiencer is an inexperienced worker, calculates a difference, and calculates a proficiency level corresponding to the difference based on a predetermined reference value; an external input device for creating and editing content; The processing device is provided with a large-capacity storage device that stores work equipment and work tools to be displayed in the virtual experience, The work equipment and the work tools are displayed in the virtual space when called by the skilled worker during the virtual experience, A work training support system characterized in that the teacher data is created by editing and creating a work model by a content editor who accesses the training support system via the external input device.

2. The work education support system according to claim 1, characterized in that the measuring device includes a target or camera for detecting the movements of the virtual experiencer, and a sensor for detecting the line of sight of the virtual experiencer or a physical quantity based on the movements.

3. The work training support system according to claim 1 or 2, characterized in that the display device includes an HMD (Head Mounted Display) worn by the virtual experiencer, and various operational instructions are shown to the virtual experiencer via the HMD.

4. The work training support system according to any one of claims 1 to 3, characterized in that when the proximity distance between objects in the display of the field of view image becomes equal to or less than a predetermined value, the image is displayed in a manner that expresses an appropriate joining state defined between the objects.

5. a display device that displays a field of view image in a virtual space; a processing device that outputs the field of view image to the display device, and records the physical quantity data and gaze movement data as training data if the virtual user is a skilled worker, and compares the physical quantity data and gaze movement data with the training data if the virtual user is an unskilled worker, calculates a difference, and determines a proficiency level corresponding to the difference based on a predetermined reference value; and an external input device, said content creation program being applicable to a work training support system that enables unskilled workers engaged in on-site work to learn the skills of skilled workers, said content creation program comprising: a measuring device that detects physical quantity data and gaze movement data of a virtual user performing a specific task; The processing device stores the work equipment and work tools to be displayed in the virtual experience, and a content editing program that is activated by accessing the external input device, When the skilled worker calls for a virtual experience, the work equipment and the work tools are displayed in the virtual space, and creating a video of work experience content by selecting and adding additional information about the unskilled worker recorded in the processing device to a video taken during the virtual experience by the skilled worker via the content editing program; A content creation program for an occupational education support system, characterized in that it is capable of executing a process of converting an image of the occupational experience content into a DataSheet according to the program and outputting it.

Images