Control device and information presentation method

Abstract

To improve behavior change and work efficiency of a worker by providing content suitable for a level of the worker.SOLUTION: A control device which presents information to a worker includes a computer that includes a calculation device that executes predetermined calculation processing and a storage device accessible by the calculation device. The calculation device includes an acquisition unit that acquires work information of the worker. The calculation device includes a work process management unit that estimates a work process and a work speed executed by the worker from the work information, changes at least one of a reproduction speed and a reproduction position to precede the work of the worker, and creates content for presenting information on the work of the next process. The calculation device includes an output unit that outputs the created content.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device that presents information to a worker. [Background technology]

[0002] In the past, paper manuals or manuals displayed on tablet devices have been widely used to instruct field workers on their work. Recently, there have been efforts to use head-mounted displays and MR technology to instruct workers on work by showing them a demonstration video superimposed on the real world.

[0003] The following prior art exists as a system for understanding work conditions using a virtual space. Patent Document 1 (JP 2021-47610 A) describes a situation understanding support system in which a worker wearing an MR-HMD observes a construction object within a space that is a construction site from various positions and directions, and a terminal device measures the three-dimensional shape of the construction object from images captured by the MR-HMD. The terminal device receives three-dimensional shape data representing the three-dimensional shape of the construction object, and generates an image in which an input field for inspection results related to the construction of the construction object is superimposed on the three-dimensional shape of the construction object as seen by the inspector in a virtual space that shares a common space and coordinate system and is determined based on the three-dimensional shape data and the position and orientation of the VR-HMD worn by the inspector, and displays the image on the VR-HMD. The system describes a situation understanding support system in which the inspector enters the results of the inspection conducted into the input field while viewing the three-dimensional shape of the construction object displayed on the VR-HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2020-144233 Summary of the Invention [Problem to be solved by the invention]

[0005] The situation understanding support system described in the aforementioned Patent Document 1 dynamically changes the content of the video display depending on the trainee's stress level, but there is no mechanism for specifying the display order or hiding of multiple contents, making it difficult to provide content appropriate to the level of the worker, and making it difficult to change the worker's behavior and improve work efficiency.

[0006] The present invention aims to change the behavior of workers and improve work efficiency by providing content that is suited to the level of the workers. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a control device for presenting information to a worker is configured by a computer including an arithmetic unit that executes predetermined arithmetic processing and a storage device accessible by the arithmetic unit, wherein the arithmetic unit has an acquisition unit that acquires work information of the worker, a work process management unit that estimates the work process and work speed being performed by the worker from the work information and changes at least one of the playback speed and playback position so that the work is performed ahead of the work of the worker, and creates content that presents information about the work of the next process, and an output unit that outputs the created content. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to encourage behavioral changes in workers and improve work efficiency. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration of a work support system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the physical configuration of the control device of the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the logical configuration of the control device of the present embodiment. [Figure 4]FIG. 10 is a diagram showing an example of a process list according to the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a structured database according to the present embodiment. [Figure 6] 3 is a flowchart of a process executed by a control device of the present embodiment. [Figure 7] 10 is a timing chart showing an example of changing the playback speed in this embodiment. [Figure 8] 10 is a timing chart showing an example of changing the playback start position in this embodiment. [Figure 9] FIG. 10 is a logical block diagram of a control device and its peripherals according to a second embodiment. [Figure 10] 10 is a table showing an example of a process list according to a second embodiment. [Figure 11] 10 is a flowchart of a process executed by a control device according to a second embodiment. [Figure 12] 10 is a flowchart of a process executed by a control device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Example 1 FIG. 1 is a diagram showing the configuration of a work support system according to an embodiment of the present invention.

[0011] The work support system of this embodiment includes an imaging device 1, an edge processing device 2 connected to the imaging device 1, a control device 10 that processes the observation results by the imaging device 1, a network 4 that connects the edge processing device 2 to the control device 10, and an MR device 5. An operation target device 3 that is operated by a wearer of the MR device 5 may be connected to the control device 10 via the network 4. The work support system may also include an administrator terminal 6.

[0012] The imaging device 1 is a sensor that observes the situation of a worksite to be shared in a virtual three-dimensional space (metaverse space) 9. The imaging device 1 may be capable of acquiring three-dimensional point cloud data. For example, a time-of-flight (TOF) camera that outputs a distance-attached image in which the distance D for each pixel is added to RGB data may be used. The imaging device 1 may also be a stereo camera equipped with two complementary metal oxide semiconductor (CMOS) image sensors, a structured light sensor that combines a projection pattern light-emitting element and an image sensor, or a sensor device that combines a distance sensor and a simple RGB camera to adjust the relationship between pixels. Furthermore, a sensor equipped with a function for estimating distance information for each pixel from an RGB image using machine learning or the like may be used. Multiple imaging devices 1 may be installed to cover a wide area of ​​the worksite, including the worker's work area, and the observation ranges of each imaging device 1 may be installed so that they overlap. The imaging device 1 observes static objects, whose shape and position do not change, such as equipment installed on the site and room structures, as well as dynamic objects, whose shape and position change, such as vehicles, construction machinery, robots, workers, tools, and work targets. Furthermore, the image capturing device 1 may be a camera that captures an image of a worker that can be motion captured by the control device 10.

[0013] The edge processing device 2 is a computer that generates 3D sensing data including multiple three-dimensional plane data and a human skeletal model from the point cloud data acquired by the imaging device 1. The edge processing device 2 generates 3D sensing data from the point cloud data, thereby reducing the amount of communication between the edge processing device 2 and the control device 10 and preventing congestion on the network 4. Note that if there is no problem with the bandwidth of the network 4, the point cloud data may be transmitted directly to the control device 10 and then three-dimensional information may be generated.

[0014] The control device 10 is a computer that is provided on the network 4 and realizes edge computing, and in this embodiment, it saves three-dimensional information collected from one or more edge processing devices 2 as a work log and generates a virtual three-dimensional space 9 from the three-dimensional information. Note that the control device 10 may estimate the movements of the worker by motion capture processing using images captured by the imaging device 1.

[0015] The task support system of this embodiment generates a virtual three-dimensional space 9 from three-dimensional information in the following procedure.

[0016] First, the imaging device 1 observes the situation at the site and transmits the observed point cloud data to the edge processing device 2. Then, the edge processing device 2 generates three-dimensional information including the point cloud data and three-dimensional model data observed by the imaging device 1. For example, the edge processing device 2 integrates the point cloud data observed by the multiple imaging devices 1 based on the relationship between the positions and observation directions of the multiple imaging devices 1.

[0017] The edge processing device 2 then executes a high-speed 3D modeling process for static objects that configures the outer surfaces of static objects using an algorithm for generating surfaces based on the positional relationships of adjacent point clouds. It also extracts ranges where shape and position change from the point cloud data, generates a skeletal model obtained by skeletal estimation, and executes a high-speed 3D modeling process for dynamic objects that models a person.

[0018] Thereafter, the edge processing device 2 segments the three-dimensional model by distinguishing between dynamic and static objects and determining the range that is meaningful as an object according to the continuity of the constructed surfaces and the range of dynamic objects.

[0019] The control device 10 also generates an avatar of the site worker from the skeleton model of the recognized dynamic object, and generates a virtual three-dimensional space 9 by mapping the generated avatar and three-dimensional model data of the recognized static object.

[0020] The control device 10 recognizes the segmented 3D model and identifies the object. For example, the type of object can be estimated using a machine learning model that has learned from images of the object installed at the site or a model that records the 3D shape of the object installed at the site.

[0021] The control device 10 recognizes the type of worker's behavior from motion data including the worker's movements and position at the site represented by the skeletal model. For example, the worker's behavior can be estimated using motion data based on past changes in the worker's skeletal model and a machine learning model learned from the worker's behavior.

[0022] Furthermore, the control device 10 recognizes the work of the worker by combining the identified object and the recognized behavior of the worker. For example, the work of the worker can be estimated by a machine learning model learned from objects and behaviors, or a knowledge graph that associates objects and behaviors.

[0023] The control device 10 records the recognized work in the structured database 18. In the structured database 18, the object and action used to recognize the work, and the motion data resulting from the change in the skeletal model during the action are registered as related information.

[0024] The network 4 is a wireless network suitable for data communication that connects the edge processing device 2 and the control device 10, and may be, for example, a high-speed, low-latency 5G network. Note that if the edge processing device 2 is installed in a fixed location, a wired network may also be used.

[0025] The MR device 5 is worn by a worker who operates the target device 3 on-site. The MR device 5 includes a processor that executes programs, a memory that stores programs and data, a network interface that communicates with the control device 10, and a display that displays images transmitted from the control device 10. The display may be transparent, allowing the wearer to view the surroundings through the display, superimposed on the image transmitted from the control device 10. The MR device 5 may also include a camera that captures a front view of the wearer and transmits the image captured by the camera to the control device 10. The MR device 5 may also display an image captured by a camera that captures a front view of the wearer, superimposed on the image transmitted from the control device 10. The MR device 5 may also include a camera that captures the wearer's eyes and detect the wearer's line of sight from the image captured by the camera. The MR device 5 may also include a microphone that detects sounds the wearer is hearing. The worker can view information related to the work displayed on the MR device 5 and share the virtual three-dimensional space 9.

[0026] The worker may also wear a wearable sensor (for example, a tactile glove). The tactile glove detects the worker's sense of touch and transmits it to the control device 10. The wearable sensor may also detect the movements of the worker's fingers, and the control device 10 may generate a skeletal model of the worker from the movements of the fingers detected by the wearable sensor and detect the worker's behavior.

[0027] The manager terminal 6 is a computer used by a site manager who uses the work support system, and can display the progress of the work of the workers and information of the virtual three-dimensional space 9 (for example, an overhead image).

[0028] The work support system of this embodiment may have a cloud 8 that forms a large-scale virtual three-dimensional space for sharing three-dimensional information collected from multiple control devices 10. The large-scale virtual three-dimensional space formed in the cloud 8 is an integration of the virtual three-dimensional spaces formed by the multiple control devices 10, and can form a large-scale virtual three-dimensional space over a wide area.

[0029] Access to the control device 10 from the MR device 5 and the administrator terminal 6 may be authenticated by ID and password or by the unique address (for example, MAC address) of these devices to ensure security of the work support system.

[0030] 2 is a block diagram showing the physical configuration of a computer provided in the work support system of this embodiment. In FIG. 2, a control device 10 is shown as an example of a computer, but the edge processing device 2 and the administrator terminal 6 may also have the same configuration.

[0031] The control device 10 of this embodiment is configured by a computer having a processor (CPU) 101, a memory 102, an auxiliary storage device 103, and a communication interface 104. The control device 10 may also have an input interface 105 and an output interface 106.

[0032] The processor 101 is a computing device that executes programs stored in the memory 102. The processor 101 executes various programs to realize various functional units of the control device 10 (e.g., a user work information acquisition unit 11, a work process management unit 12, a communication unit 16, a content output unit 19, etc.). Note that some of the processing performed by the processor 101 by executing the programs may be executed by another computing device (e.g., hardware such as a GPU, ASIC, or FPGA).

[0033] The memory 102 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS), etc. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 101 and data used when the programs are executed.

[0034] The auxiliary storage device 103 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 103 also stores data used by the processor 101 when executing a program and the program executed by the processor 101. That is, the program is read from the auxiliary storage device 103, loaded into the memory 102, and executed by the processor 101 to realize each function of the control device 10.

[0035] The communication interface 104 is a network interface device that controls communication with other devices (for example, the edge processing device 2, the cloud 8) in accordance with a predetermined protocol.

[0036] The input interface 105 is an interface to which input devices such as a keyboard 107 and a mouse 108 are connected and which receives input from an operator. The output interface 106 is an interface to which output devices such as a display device 109 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by the user. Note that an administrator terminal 6 connected to the control device 10 via a network may provide the input and output devices. In this case, the control device 10 may have a web server function, and the administrator terminal 6 may access the control device 10 using a predetermined protocol (for example, http).

[0037] The program executed by the processor 101 is provided to the control device 10 via removable media (CD-ROM, flash memory, etc.) or a network, and is stored in a non-volatile auxiliary storage device 103, which is a non-transitory storage medium. For this reason, the control device 10 preferably has an interface for reading data from removable media.

[0038] The control device 10 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0039] FIG. 3 is a logical block diagram of the control device 10 of this embodiment.

[0040] The control device 10 has a user work information acquisition unit 11, a work process management unit 12, a communication unit 16, a structured database 18, and a content output unit 19. The work process management unit 12 has an analysis unit 13, a presented content creation unit 14, a content database 15, and a process list 17.

[0041] The user task information acquisition unit 11 acquires task information of the worker from the imaging device 1 and the MR device 5. The task information is information about the worker's tasks, such as finger movements, hand positions, head positions, and line of sight.

[0042] The analysis unit 13 analyzes the work information and estimates the work process and work speed. The content database 15 stores content to be presented to the worker (for example, work instructions such as a manual). The presented content creation unit 14 creates content to be presented to the worker. In this embodiment, the content read from the content database 15 is processed to create content to be presented to the worker. The communication unit 16 controls communication with other devices (for example, the edge processing device 2). The process list 17 records the actions of the worker in the work process. The structured database 18 records the analyzed work of the worker. The content output unit 19 outputs the content created by the presented content creation unit 14 and presents it to the worker.

[0043] The content stored in the content database 15 includes text information that indicates information about the work using text, audio information that indicates information about the work using audio, pointer information that indicates a position related to the work, still image information that indicates information about the work using still images, and video information that indicates information about the work using video images.

[0044] It is preferable that the content stored in the content database 15 be created so that it is clear whether the model worker shown in the content is touching the work object. For example, if the worker is touching the work object, a mark indicating the contact may be displayed, or the contact area (for example, the hand, the work object, or both) may be displayed in a different color. By creating the content in this way, it is possible to clearly instruct the work content even in an image taken from an angle where it is difficult to tell whether the worker is touching the work object, such as when the worker's hands are not visible.

[0045] FIG. 4 is a diagram showing an example of the process list 17 of this embodiment.

[0046] The process list 17 records data on major processes, minor processes, head positions, hand positions, line of sight, work content, and complexity in association with each other. Although the process list 17 is shown in list format in Fig. 4, it may be configured in other data structures.

[0047] The major process and the sub-process are identification information for uniquely identifying the process, and the major process corresponds to the "process" described later in Figures 7 and 8, and the sub-process corresponds to the subdivided sub-process described later in Figures 7 and 8. The head position, hand position, and gaze are the standard movements of the worker in the process and correspond to the motion data of the worker acquired by the analysis unit 13 as work information. The head position and hand position are analyzed from the image captured by the imaging device 1, and the gaze is analyzed from the gaze direction acquired by the MR device 5 and objects present in the gaze direction. The work content is the work content of the process. The complexity is the complexity of the process and may be determined, for example, by the number of movements in the process, the amount of hand movement, or the success or failure rate of the work.

[0048] The process list 17 is referenced to derive the target variables, ie, the major process, minor process, work content, and complexity, using the head position, hand position, and line of sight as explanatory variables.

[0049] FIG. 5 is a diagram showing an example of the structure of the structured database 18 of this embodiment.

[0050] The structured database 18 records the work analyzed by the analysis unit 13. Although the structured database 18 is shown in table format in Fig. 5, it may be configured in another data structure.

[0051] The structured database 18 includes pre-recorded task-related information 181 and task acquisition information 182 acquired in accordance with the actions of the worker.

[0052] The work-related information 181 stores a work ID, a work reference time, a work manual, work video content, and work text content in association with each other. The work ID is identification information of the work recorded in advance. The work reference time is the standard time for the work performed by the worker. The work manual is an instruction manual for the work performed by the worker, and link information for accessing the instruction manual may be recorded. The work video content is a video of the work performed by the worker, performed by an expert or previously performed by the worker, and identification information for the video and link information for accessing the video may be recorded. The work text content is text information related to the work performed by the worker, and identification information for the text information and link information for accessing the text information may be recorded.

[0053] The task acquisition information 182 stores, in association with each other, an action ID, actual task time, environmental objects, worker motion, worker position, worker viewpoint, worker sound field, worker tactile sensation, worker vital signs, worker proficiency, task ID, and task log. The action ID is identification information for an action, which is a series of movements of a worker. The actual task time is the time required for the worker's action. The environmental objects are objects (e.g., a room, a floor, equipment, tools, screws) photographed in relation to the worker's action. The worker motion is time-series data of the coordinates of the feature points of the worker's skeletal model (fingers, joints of arms, etc., and head). The worker position is the positions of the worker's feature points (head, left and right hands, etc.) and their positional relationship (distance, direction) with environmental objects. The worker viewpoint is the worker's line of sight and the intersection of the line of sight and the surface of an object in the line of sight. The worker sound field is the sound heard by the worker, and link information for accessing sound data may be recorded. The worker's tactile sense is the worker's tactile sense obtained using tactile gloves. The worker's vital signs include the worker's voice, facial expression, and pulse rate estimated from changes in blood flow, and are used to estimate the worker's emotions and attributes. The worker's proficiency is the worker's proficiency detected by the proficiency detection process. The task ID is the worker's task recognized by the task recognition process. The task log is the result of the task, and records whether it was completed successfully, re-tasked, or abnormally.

[0054] FIG. 6 is a flowchart of the process executed by the control device 10 of this embodiment.

[0055] First, the user work information acquisition unit 11 acquires work information (S11).

[0056] Next, the analysis unit 13 refers to the process list 17, analyzes the acquired work information, and estimates the work process and work speed performed by the worker (S12). For example, the work process performed by the worker may be estimated based on the degree of correspondence between motion data acquired as work information, such as finger movements, hand position, head position, and gaze, and the standard movements of the work process recorded in the predetermined process list 17. The work process may also be estimated based on the worker's specific movements. For example, if the worker performs a pointing and checking movement and the worker's gaze position is at a location that should be checked in that process, it may be determined that the pointing and checking is correct, and that the process has been completed and that the worker should proceed to the next process. The work process may also be estimated from the work information using a machine learning model that has learned the worker's movements and work processes. Furthermore, the process may be estimated by detecting the worker's operation of a physical or on-screen switch. Furthermore, the ratio between the estimated process completion timing and the standard time for each process is calculated to calculate the worker's work speed.

[0057] Next, the work process management unit 12 determines whether it is time to switch content (S13). As shown in Figs. 7 and 8, a content switching timing (CP) is defined for the video content presented to the worker, and it is determined whether the video currently being played has reached the content switching timing. Note that the period from a predetermined time (for example, 2 seconds) before the content switching timing to the end of the content may be determined to be the content switching timing. Furthermore, a period during which no content is being played may also be determined to be the content switching timing, since playback of new content can be started. Furthermore, the timing at which the analysis unit 13 estimates that a major process has ended may be determined to be the content switching timing.

[0058] If it is determined that it is not time to switch the content, then in step S16, the presented content creator 14 continues playing the content that is currently being played.

[0059] When it is determined that it is time to switch content, the presented content creation unit 14 refers to the work-related information 181 in the structured database 18 to identify content to be presented to the worker in the next work process, and reads the identified content from the content database 15 (S14). When the presented content creation unit 14 selects content to be presented so as to include at least two of text information, audio information, pointer information, still image information, and video information, it is possible to present information to the worker in an easy-to-understand manner.

[0060] Then, the presented content creation unit 14 determines the playback speed and playback position of the content based on the user's settings and the progress of the worker's work (S15). In this way, the presented content creation unit 14 creates content that switches to scenes representing each process at the content switching timing.

[0061] Then, the presented content creating unit 14 transmits the content to the MR device 5, and presents the content with the adjusted playback speed and playback position to the worker (S16).

[0062] FIG. 7 is a timing chart showing an example of changing the playback speed so as to present information about a task that precedes the progress of the task in this embodiment.

[0063] Figure 7 shows the actual work progress, the content presented to the worker, and the standard time for the work process according to the progress of the work process. In addition, a content switching timing CP is set at the end timing of the work process or content.

[0064] Since the first process A is completed in a shorter time than the standard time, the content of process A is terminated at the content switching timing CP1 when process A is completed, and the content of the next process B is presented at a slightly faster speed (for example, 1.1 times faster).

[0065] The next process B was slightly longer than the standard time divided by 1.1, but was completed in less time than the standard time, so from the content switching timing CP2 when process B ended, the content of the next process C is presented at the same speed as process B (for example, 1.1 times faster).

[0066] In the next process C, the content of process C is displayed at standard time divided by 1.1 (CP3), but because process C is not yet finished, the content of process C is presented again at 1.0 times the standard time. The content that is presented again due to the end of the process should be presented from the optimal position midway through process C (for example, the position where the work is currently being done) based on work process analysis. In this way, by re-presenting the content of the current process, the cause of the delay in the process (for example, the work has stopped because it is not going well) can be eliminated, and the work can proceed.

[0067] After the process C is completed, the next process D is started (CP4). In the process D, since the process C took longer than the standard time, the content of the process D is presented at a slightly slower speed (for example, 0.9 times the normal speed).

[0068] Since process D was completed in a shorter time than the standard time, the content of process D is terminated at the content switching timing CP5 when process D is completed, and the content of the next process E is presented at a slightly faster speed (for example, 1.1 times faster).

[0069] In this embodiment, the content playback speed for each process may be increased or decreased in predetermined steps. In this case, the content playback speed may be increased until the working speed is slower than the playback speed, and then decreased until the working speed is faster than the playback speed.

[0070] The playback speed of the content may also be determined according to the shortening or lengthening rate of the previous process. For example, if the working speed is 0.8 times the playback speed, the content of the next process may be played at 1.25 times the speed, and if the working speed is 1.25 times the playback speed, the content of the next process may be played at 0.8 times the speed.

[0071] The playback speed may also be determined based on the complexity of the work content. For example, content for a highly complex process C recorded in the process list 17 may be presented at 0.9x slower than normal speed, while content for a less complex process A may be presented at 1.1x faster than normal speed. Furthermore, complexity may be used as a coefficient to multiply the worker's work speed. For example, in the example above, process B is played back at 1.1x speed, but the content for process B may be presented at 1.21x speed by multiplying it by the coefficient of complexity A, 1.1.

[0072] In Figure 7, the content presentation speed is changed for each process, but it is also possible to subdivide the process into multiple small processes, set checkpoints (not shown) at the boundaries of the small processes (i.e., in the middle of a large process), and determine at each checkpoint whether the work is faster or slower than the standard time, and then change the content presentation speed and playback start position.

[0073] FIG. 8 is a timing chart showing an example of changing the playback start position so as to present information about a task that precedes the progress of the task in this embodiment.

[0074] 8 shows the actual work progress and the content presented to the worker according to the progress of the work process. In addition, a content switching timing CP is set at the timing when the work process or content ends.

[0075] The presentation of the content for the next process B begins at content switching timing CP1, which is set at a predetermined time (Δt) when the first process A ends. The predetermined time (Δt) that determines the timing to start presenting the content is preferably 0.5 to 3 seconds. In this way, by starting to present the content for a process before the process starts, information about the next process can be obtained in advance, allowing for a smooth transition to the next process.

[0076] In Figure 8, the timing at which the content presentation starts is changed for each process, but it is also possible to subdivide the process into multiple small processes, set checkpoints (not shown) at the boundaries of the small processes (i.e., in the middle of a large process), and determine at each checkpoint whether the work is ahead or behind the standard time, and change the playback position and presentation speed of the content.

[0077] In this way, in the form shown in Figure 7, the playback speed of the content for the next process is adjusted according to the speed at which the worker is working, and in the form shown in Figure 8, the playback start position of the content for the next process is adjusted according to the speed at which the worker is working, and information about the work that precedes the progress of the work by the worker is presented.

[0078] FIG. 7 shows an example of changing the playback speed, and FIG. 8 shows an example of changing the playback start position, but both the playback speed and the playback start position may be changed to present task information in advance.

[0079] As described above, according to the first embodiment of the present invention, by presenting a playback speed faster than the worker's work or a process further ahead than the actual work, it is possible to encourage behavioral change in the worker and improve work efficiency.

[0080] <Example 2> In the second embodiment, the process being performed by the worker is estimated by also referring to the environmental information of the virtual three-dimensional space 9. In the second embodiment, differences from the first embodiment will be mainly described, and descriptions of the same configurations and functions as those in the first embodiment will be omitted.

[0081] FIG. 9 is a logical block diagram of the control device 10 and its periphery according to this embodiment.

[0082] The control device 10 is connected to the operation target device 3 and the imaging device 1 so as to be able to communicate with them. The control device 10 has a user work information acquisition unit 11, a work process management unit 12, a communication unit 16, a structured database 18, a content output unit 19, and a user environment information acquisition unit 20. The work process management unit 12 has an analysis unit 13, a presented content creation unit 14, a content database 15, and a process list 17.

[0083] The user work information acquisition unit 11, work process management unit 12, presented content creation unit 14, content database 15, communication unit 16, structured database 18, and content output unit 19 are the same as those in the control device 10 of the first embodiment described above. The user environment information acquisition unit 20 acquires information on the status of objects and people other than the worker, which is accumulated as environmental information in the virtual three-dimensional space 9. For example, it acquires the operation history and status of the operation target device 3 and information on the site photographed by the imaging device 1. The analysis unit 13 analyzes the work information from the worker's movements and environmental information, and estimates the work process and work speed.

[0084] The operation target device 3 has an internal state calculation unit 300, a measuring device 301, and a communication unit 302. The measuring device 301 measures the operating state of the operation target device 3. The internal state calculation unit 300 derives the state of the operation target device 3 from the measurement results by the measuring device 301, the operation state of the operation panel, etc. The communication unit 302 controls communication with other devices (for example, the control device 10).

[0085] FIG. 10 is a diagram showing an example of the process list 17 of this embodiment.

[0086] The process list 17 records data on major processes, minor processes, head positions, hand positions, line of sight, work content, and environmental information in association with each other. The process list 17 of the second embodiment may also record complexity data. While the process list 17 is shown in list format in Fig. 10, it may also be configured with other data structures.

[0087] The major process, minor process, head position, hand position, line of sight, and work content are the same as those in the process list 17 of the above-described embodiment 1. The environmental information is information about the operation target device 3 on which the worker works and the operation target object, and is estimated from the video of the site captured by the imaging device 1 and the operation target device 3.

[0088] In this embodiment, the process list 17 is referenced to derive the target variables, ie, major process, minor process, work content, and complexity, using the head position, hand position, line of sight, and environmental information as explanatory variables.

[0089] FIG. 11 is a flowchart of the process executed by the control device 10 of this embodiment.

[0090] First, the user work information acquisition unit 11 acquires work information (S11).

[0091] Next, the user environment information acquisition unit 20 acquires information on the states of objects and people other than the worker, which is accumulated as environmental information of the virtual three-dimensional space 9 (S100). As described above, the environmental information is, for example, the operation history and state of the operation target device 3, and information on the site photographed by the imaging device 1.

[0092] Next, the analysis unit 13 refers to the process list 17, analyzes the acquired work information and environmental information, and estimates the work process being performed by the worker (S12). For example, the analysis unit 13 may estimate the work process being performed by the worker based on the degree of agreement between motion data, such as finger movements, hand position, head position, and gaze, acquired as work information, and standard actions for the work process recorded in the predetermined process list 17. Alternatively, the analysis unit 13 may estimate the work process being performed by the worker based on the operation history of the operation panel of the operation target device 3 and measurement results (e.g., current value) acquired as environmental information. The analysis unit 13 may also estimate the work process based on a specific action performed by the worker. For example, if the worker is performing a pointing and checking action and the worker's gaze position is at a location that should be checked in the process, it may be determined that the pointing and checking is correct, and that the process has been completed and that the worker should proceed to the next process. Alternatively, the work configuration may be estimated from the work information using a machine learning model that has learned the worker's movements and work processes. Furthermore, the process may be estimated by detecting the worker's operation of a physical or on-screen switch. Furthermore, the ratio of the estimated process completion timing to the standard time for each process is calculated to calculate the worker's work speed.

[0093] Next, the work process management unit 12 determines whether it is time to switch content (S13). If it is determined that it is time to switch content, the presented content creation unit 14 reads content from the content database 15 (S14) and determines the playback speed and playback position of the content (S15). Then, the presented content creation unit 14 transmits the content to the MR device 5 and presents the content with the adjusted playback speed and playback position to the worker (S16). The processing from steps S13 to S16 is the same as in the first embodiment described above.

[0094] As described above, according to the second embodiment of the present invention, in addition to the effects of the first embodiment, the accuracy of process estimation can be improved.

[0095] Example 3 In the third embodiment, the content presented to the worker is changed depending on the worker's proficiency. In the third embodiment, differences from the first embodiment will be mainly described, and descriptions of the same configurations and functions as those in the first embodiment will be omitted.

[0096] FIG. 12 is a flowchart of the process executed by the control device 10 of this embodiment.

[0097] First, the user work information acquisition unit 11 acquires work information (S11). Next, the analysis unit 13 refers to the process list 17, analyzes the acquired work information, and estimates the work process being performed by the worker (S12). Next, the work process management unit 12 determines whether it is time to switch content (S13). If it is determined that it is time to switch content, the presented content creation unit 14 reads content from the content database 15 (S14), and determines the playback speed and playback position of the content (S15). The processing from steps S11 to S15 is the same as in the first embodiment described above.

[0098] Next, the presentation content creation unit 14 determines the proficiency of the worker (S200). The proficiency of the worker may be determined using the difference between the work time of the previous process and the standard time, using past work records, or using pre-registered proficiency data.

[0099] If the proficiency level is determined to be beginner level, it is desirable to present detailed content, so the presented content creation unit 14 transmits still image content to the MR device 5 and presents it to the worker (S201), and waits for the user's response (S202). Then, when the user who has viewed the still image content inputs a response, the presented content creation unit 14 erases the still image content (S203) and presents an operation target pointer indicating the position where the worker should operate (S204). Thereafter, the presented content creation unit 14 determines the playback speed and playback position of the content (S205).

[0100] If the proficiency level is determined to be intermediate, the presented content creating unit 14 presents an operation target pointer indicating a position to be operated by the operator (S204), and then determines the playback speed and playback position of the content (S205).

[0101] Then, the presented content creating unit 14 transmits the content to the MR device 5, and presents the content with the adjusted playback speed and playback position to the worker (S16).

[0102] If the proficiency level is determined to be at the advanced level, the content needs to be presented, so the content is hidden (S206) and the process returns to step S11.

[0103] As described above, in addition to the effects of the first embodiment, the third embodiment of the present invention has the effects of reducing work errors by providing detailed content to beginners and reducing unnecessary work instructions to advanced users.

[0104] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0105] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0106] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0107] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0108] 1. Imaging device 2 Edge processing equipment 3. Device to be operated 4 Network 5. MR Devices 6. Administrator terminal 8. Cloud 9 Virtual 3D Space 10 Control device 11 User work information acquisition unit 12 Work process management department 13 Analysis Department 14 Presentation Content Creation Department 15 Content Database 16 Communications Department 17 Process List 18 Structured Database 19 Content Output Unit 20 User environment information acquisition unit 101 processors 102 memory 103 Auxiliary storage device 104 Communication Interface 105 Input Interface 106 Output Interface 107 keyboard 108 Mouse 109 Display Device 181 Work-related information 182 Work Acquisition Information 300 Internal state calculation unit 301 Measuring Equipment 302 Communications Department

Claims

1. A control device that presents information to an operator, The computer is configured with an arithmetic unit that executes predetermined arithmetic processing and a storage device that can be accessed by the arithmetic unit, The computing device includes an acquisition unit that acquires work information of a worker; a work process management unit in the computing device that estimates the work process and work speed being performed by the worker from the work information, changes at least one of the playback speed and playback position so that the work is ahead of the work of the worker, and creates content that presents information about the work of the next process; The control device is characterized in that the arithmetic device has an output unit that outputs the created content.

2. The control device according to claim 1, The control device is characterized in that the work process management unit creates content whose playback speed is adjusted in accordance with the work speed of the worker.

3. The control device according to claim 1, The control device is characterized in that the work process management unit creates content that is configured by switching scenes over time.

4. The control device according to claim 1, the acquisition unit acquires environmental information around the worker, The control device is characterized in that the work process management unit estimates the work process and work speed being performed by the worker from the work information and the environmental information, and changes at least one of the playback speed and playback position so that the work is ahead of the worker's work, thereby creating content that presents information about the work of the next process.

5. The control device according to claim 4, The control device, wherein the environmental information includes information about an object on which the worker is working.

6. The control device according to claim 1, The control device is characterized in that the work process management unit creates content that presents information about the next process of work at at least one of a playback speed faster than the estimated work speed and a speed that precedes the work of the worker.

7. The control device according to claim 1, The control device is characterized in that the work process management unit creates content for a next process in a series of work processes from the estimated work process.

8. The control device according to claim 7, The control device is characterized in that the work process management unit creates content so that it includes at least two of text information, audio information, pointer information, still image information, and video information.

9. The control device according to claim 7, The control device is characterized in that the work process management unit determines the proficiency level of the worker based on the work time of each work process.

10. The control device according to claim 9, The control device is characterized in that the work process management unit does not display content when it is determined that the worker's proficiency level is high.

11. The control device according to claim 1, The control device is characterized in that the acquisition unit acquires at least one of an image captured by an imaging device and an operation input by a user as work information of the worker.

12. The control device according to claim 1, The control device is characterized in that the work process management unit creates content for the next process that presents information about the work of the worker 0.5 to 3 seconds ahead of the work of the worker.

13. The control device according to claim 1, The control device is characterized in that the work process management unit determines that the process has been completed if the worker is performing a pointing check action and the worker's line of sight is at a location that should be checked in the work process being performed by the worker.

14. An information presentation method in which a control device presents information to a worker, the control device is configured by a computer having an arithmetic unit that executes predetermined arithmetic processing and a storage device that can be accessed by the arithmetic unit; The information presentation method includes: an acquisition step in which the arithmetic device acquires work information of a worker; a work process management procedure in which the computing device estimates the work process and work speed being performed by the worker from the work information, and changes at least one of the playback speed and playback position so that the work is ahead of the work of the worker, thereby creating content that presents information about the work of the next process; an output step in which the computing device outputs the created content.

Images