Control apparatus and information presentation method

Abstract

To improve work efficiency of a worker by acquiring work information of the worker, detecting a divergence from a model operation or state, and dynamically controlling content.SOLUTION: A control apparatus 8 presents a three-dimensional operation to a worker using cross reality, the control apparatus including a work information acquisition unit 10, a calculation unit 11, and a communication unit 15. The calculation unit includes a control unit 12, an output unit 13, and a content database 14. The work information acquisition unit acquires work information of a worker from an imaging apparatus or an MR device 5. The work information is information on an operation of a worker. The control unit analyzes the work information of the worker, calculates a spatial divergence between actual work and model work included in consecutive instruction content indicating a model operation and state, and dynamically controls a display method for the consecutive instruction content based on a calculation result. The content database stores the consecutive instruction content to be presented to the worker. The communication unit controls communication with another apparatus.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a control device that presents information to an operator. [Background technology]

[0002] Conventionally, paper manuals or manuals displayed on tablet devices have been widely used as methods for instructing on-site workers on work content. Recently, there have been efforts to instruct on work content by using head-mounted displays and XR (cross reality) technology to check images of work procedures superimposed on the real world.

[0003] The following prior art is a system for instructing work content using an augmented reality space. Patent Document 1 (JP Patent Publication No. 2023-16589) describes an inspection work support system that creates a sequence code image indicating the order of inspection work based on a camera that captures an image in front of the inspector's face, an inspector display device for displaying the image, a display device for displaying the image, and inspection work order information extracted by a database access means, and changes the position of this sequence code image so that it appears as if it were attached to the corresponding inspection target work location of the inspection target work location in the current image of the inspection target device identified by the inspection target work location identification means, thereby creating a display sequence code image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-16589 Summary of the Invention [Problem to be solved by the invention]

[0005] In such a work system, the worker may miss the content that instructs the worker on the work. This can happen because the content is displayed outside the viewing angle of the MR device, or because the worker's field of view moves away from the content when performing a task other than the content. In addition, when the content moves dynamically, this can also occur when the worker is unable to keep up with the speed of the content's movement. In such cases, it is necessary to provide assistance to bring the content back into the field of view.

[0006] The inspection work support system described in the above-mentioned Patent Document 1 stops displaying the display sequence code image when it is detected that the inspector's eye gaze is off the inspection target work location beyond a predetermined distance. However, the inspection target work location is not clearly indicated to the inspector, and there is a problem that the inspector may lose sight of the inspection target work location. In addition, since it does not handle three-dimensional content, there is a problem that it is difficult to give work instructions for complex shapes and actions.

[0007] The present invention aims to improve the work efficiency of a worker by acquiring work information of the worker, detecting deviations from model actions and states, and dynamically controlling the playback of content. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a control device that utilizes cross reality to present three-dimensional actions to a worker is configured by a computer including a calculation device that executes a predetermined calculation process and a storage device accessible by the calculation device, the calculation device has an acquisition unit that acquires work information of a work being performed by the worker, the calculation device has a calculation unit that executes a calculation for presenting continuous teaching content corresponding to the work being performed by the worker to the worker, and the calculation device has an output unit that outputs content, and the calculation unit has a control unit that dynamically controls a display method of the continuous teaching content according to a degree of spatial separation between the work information acquired by the acquisition unit and the continuous teaching content. Effect of the Invention

[0009] According to one aspect of the present invention, the work efficiency of a worker can be improved by acquiring work information of the worker, detecting deviation from an ideal operation and state, and dynamically controlling the content. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of a work support system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a physical configuration of a computer provided in the work support system of the first embodiment. [Diagram 3] FIG. 2 is a logical block diagram of a control device according to the first embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of work information according to the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of work information according to the first embodiment. [Figure 6A] FIG. 11 is a diagram showing an example of calculating a spatial deviation degree from a continuous teaching content in the first embodiment. [Figure 6B] FIG. 11 is a diagram showing an example of calculating a spatial deviation degree from a continuous teaching content in the first embodiment. [Figure 6C] FIG. 11 is a diagram showing an example of calculating a spatial deviation degree from a continuous teaching content in the first embodiment. [Figure 6D] FIG. 11 is a diagram showing another example of calculating the spatial deviation degree from the continuous teaching content in the first embodiment. [Figure 7A] FIG. 11 is a diagram showing an example of calculating a spatial deviation between a worker and continuous teaching content according to the first embodiment. [Figure 7B] FIG. 11 is a diagram showing an example of calculating a spatial deviation between a worker and continuous teaching content according to the first embodiment. [Figure 7C] FIG. 11 is a diagram showing an example of calculating a spatial deviation between a worker and continuous teaching content according to the first embodiment. [Figure 8]11 is a diagram for explaining whether a feature point of the continuous learning content of the first embodiment is within the visual field of the worker. FIG. [Figure 9A] 10A to 10C are diagrams for explaining an example of the relationship between feature points and the field of view of the continuous learning content in the first embodiment. [Figure 9B] 10A to 10C are diagrams for explaining an example of the relationship between feature points and the field of view of the continuous learning content in the first embodiment. [Figure 9C] 10A to 10C are diagrams for explaining an example of the relationship between feature points and the field of view of the continuous learning content in the first embodiment. [Figure 10] 4 is a flowchart of a process executed by a control device according to the first embodiment. [Figure 11] 11 is a flowchart of a process executed by a control device according to a second embodiment. [Figure 12] FIG. 11 is a logical block diagram of a control device according to a third embodiment. [Figure 13] 11 is a flowchart of a process executed by a control device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Example 1> FIG. 1 is a diagram showing a configuration of a work support system according to a first embodiment of the present invention.

[0012] The work support system of this embodiment includes an imaging device 1, an edge processing device 2 connected to the imaging device 1, a control device 8 that processes the observation results by the imaging device 1, a network 4 that connects the edge processing device 2 to the control device 8, and an MR device 5. An operation target device 3 operated by a wearer of the MR device 5 may be connected to the control device 8 via the network 4. The work support system may also include an administrator terminal. The work support system may also be configured to be connected to a cloud 7 via the external Internet 6. The work log and work analysis results acquired by the control device 8 are stored in the control device 8 or the cloud 7.

[0013] The imaging device 1 is a sensor that observes the situation at the site. The imaging device 1 may be capable of acquiring three-dimensional point cloud data, and may be, for example, a TOF camera that outputs an image with distances in which RGB data is assigned a distance D for each pixel. The imaging device 1 may be a stereo camera equipped with two CMOS (Complementary Metal Oxide Semiconductor) image sensors, a structured light type sensor that combines a projection pattern light emitting element and an image sensor, or a sensor device that combines a distance sensor and a simple RGB camera to adjust the relationship between pixels. Furthermore, a sensor that is equipped with a function of estimating distance information for each pixel from an RGB image using machine learning or the like may be used. A plurality of imaging devices 1 may be provided to cover a wide range of the site including the work range of the worker, and the imaging devices 1 may be installed so that the observation ranges of the imaging devices 1 overlap. The imaging device 1 observes static objects, such as equipment installed at the site and structures in a room, whose shapes and positions do not change, and dynamic objects, such as vehicles, construction machinery, robots, workers, tools, and work objects, whose shapes and positions change, as objects. Furthermore, the imaging device 1 may be a camera that captures an image of a worker for which the control device 8 can perform motion capture processing.

[0014] The edge processing device 2 is a computer that generates 3D sensing data including a plurality of three-dimensional plane data and a human skeletal model from the point cloud data acquired by the imaging device 1. The edge processing device 2 generates 3D sensing data from the point cloud data, thereby reducing the amount of communication between the edge processing device 2 and the control device 8 and suppressing congestion on the network 4. Note that, if there is no problem with the bandwidth of the network 4, the point cloud data may be transmitted as is to the control device 8 and then the three-dimensional information may be generated.

[0015] The control device 8 receives the work information of the worker from the MR device 5, or detects the work information of the worker based on the three-dimensional information collected from one or more edge processing devices 2, or acquires the work information of the worker using both methods. The control device 8 may estimate the movement of the worker by a motion capture process using an image captured by the imaging device 1. The control device 8 also transmits continuous teaching content to be presented to the MR device 5. The worker can correctly understand the work content and perform it in the correct procedure by imitating the movements of the continuous teaching content displayed on the MR device 5. The continuous teaching content is a three-dimensional moving image including the three-dimensional movements of an avatar (e.g., a model) in a virtual space, and is displayed on the MR device 5 by superimposing it on the image of the worker in the real space using cross reality technology. A work is an action in which a human being performs some action, such as assembling a device, operating a device, or grasping an object.

[0016] The control device 8 may generate a virtual three-dimensional space from the three-dimensional information collected from the edge processing device 2.

[0017] The network 4 is a wireless network suitable for data communication that connects the edge processing device 2 and the control device 8, and may be, for example, a high-speed, low-latency 5G network. When the edge processing device 2 is fixedly installed, the network 4 may be a wired network.

[0018] The MR device 5 is a device worn by an operator who operates the operation target device 3 at the site, and includes a processor for executing a program, a memory for storing programs and data, a network interface for communicating with the control device 8, and a display for displaying an image transmitted from the control device 8. The display may be a transparent type so that the wearer can view the surroundings through the display superimposed on the image transmitted from the control device 8. The MR device 5 may also include a camera for capturing an image in front of the wearer, and transmit the image captured by the camera to the control device 8. The MR device 5 may also display an image captured by a camera for capturing an image in front of the wearer superimposed on the image transmitted from the control device 8. The MR device 5 may also include a sensor for detecting the head position coordinates and head posture of the wearer, and transmit the head position information and head posture (for example, the direction in which the face is facing) detected by the sensor to the control device 8. The MR device 5 may also include a sensor for detecting hand skeletal information of the wearer, and transmit the hand skeletal information detected by the sensor to the control device 8. The MR device 5 may also include a camera for capturing an image of the wearer's eyes, and detect the line of sight of the wearer from the image captured by the camera. The MR device 5 may also have a microphone to detect sounds that the wearer hears.

[0019] Furthermore, the worker may wear a wearable sensor (e.g., a tactile glove). The tactile glove detects the worker's sense of touch and transmits it to the control device 8. Furthermore, the wearable sensor may detect the movement of the worker's fingers, and the control device 8 may generate a skeletal model of the worker from the movement of the fingers detected by the wearable sensor, and detect the work information of the worker.

[0020] Access from the MR device 5 to the control device 8 may be authenticated by ID and password or by a unique address (eg, MAC address) of the device, thereby ensuring security of the work support system.

[0021] Fig. 2 is a block diagram showing the physical configuration of a computer provided in the work support system of Example 1. In Fig. 2, a control device 8 is shown as an example of a computer, but the edge processing device 2 may have the same configuration.

[0022] The control device 8 of this embodiment is configured by a computer having a processor (CPU) 101, a memory 102, an auxiliary storage device 103, and a communication interface 104. The control device 8 may also have an input interface 105 and an output interface .

[0023] The processor 101 is a calculation device that executes programs stored in the memory 102. The processor 101 executes various programs to realize each functional unit of the control device 8 (e.g., the work information acquisition unit 10, the control unit 12, the output unit 13, the communication unit 15, etc.). Note that some of the processes performed by the processor 101 by executing the programs may be executed by other calculation devices (e.g., hardware such as a GPU, ASIC, or FPGA).

[0024] The memory 102 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 101 and data used when the programs are executed.

[0025] The auxiliary storage device 103 is, for example, a large-capacity non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 103 also stores data used by the processor 101 when executing a program and the program executed by the processor 101. That is, the program is read from the auxiliary storage device 103, loaded into the memory 102, and executed by the processor 101 to realize each function of the control device 8.

[0026] The communication interface 104 is a network interface device that controls communication with other devices (for example, the edge processing device 2, the cloud 7) in accordance with a predetermined protocol.

[0027] The input interface 105 is an interface to which input devices such as a keyboard 107 and a mouse 108 are connected and which receives input from an operator. The output interface 106 is an interface to which output devices such as a display device 109 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by an operator.

[0028] The programs executed by the processor 101 are provided to the control device 8 via a removable medium (such as a CD-ROM or a flash memory) or a network, and are stored in a non-volatile auxiliary storage device 103, which is a non-transitory storage medium. For this reason, the control device 8 may have an interface for reading data from the removable medium.

[0029] The control device 8 is a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0030] FIG. 3 is a logical block diagram of the control device 8 according to the first embodiment.

[0031] The control device 8 has a work information acquisition unit 10, a calculation unit 11, and a communication unit 15. The calculation unit 11 has a control unit 12, an output unit 13, and a content database 14.

[0032] The work information acquisition unit 10 acquires work information of the worker from the imaging device 1 and the MR device 5. The work information is information about the worker's actions, such as finger movements, hand positions, head position and posture, and line of sight.

[0033] The control unit 12 analyzes the task information of the worker, calculates the spatial deviation between the model task included in the continuous teaching content that shows the model actions and states and the actual task, and dynamically controls the display method of the continuous teaching content based on the calculation result. This process will be described in detail later.

[0034] The content database 14 stores continuous teaching contents to be presented to the worker. The continuous teaching contents are composed of three-dimensional information, and are contents for teaching the work contents to the worker by presenting, for example, finger movements and head positions as moving images. In this case, it is preferable to use a human-shaped avatar or the like, since it makes the work easier to understand. Note that, in addition to the continuous teaching contents, the contents stored in the content database 14 may also include text information showing information related to the work in text, audio information showing information related to the work in audio, pointer information showing a position related to the work, still image information showing information related to the work in still images, and video information showing information related to the work in video images.

[0035] The communication unit 15 controls communication with other devices (for example, the edge processing device 2).

[0036] 4 and 5 are diagrams showing an example of the work information used by the control unit 12 in the first embodiment to calculate the degree of spatial deviation from the continuous learning content.

[0037] 4, the task information is represented by skeletal position information of the worker, and the task information is represented by three-dimensional position information of the head 300, neck 301, left shoulder 302, right shoulder 303, waist 304, right elbow 305, and right wrist 306. The control device 8 acquires the three-dimensional position information from the imaging device 1 and the MR device 5. In particular, the position information of the head 300 may utilize the head position coordinates and head pose of the worker acquired by the MR device 5. By acquiring the FOV, head position coordinates, and head pose of the MR device 5, it is possible to determine whether or not the content described below can be viewed.

[0038] 5, the task information is represented by hand skeleton information, and the task information is represented by three-dimensional position information of the avatar's fingers 40 and hand skeleton 41 performing exemplary tasks in the continuous teaching content, and the worker's fingers 42 and hand skeleton 43. As described above, the control device 8 obtains the three-dimensional position information from the imaging device 1 and the MR device 5. In particular, the hand skeleton information includes multiple pieces of position information such as the base of the wrist and each fingertip, and it is possible to obtain position information of each skeleton point during a task using the fingers, such as gripping an object or tightening a screw.

[0039] The spatial deviation degree from the continuous teaching content can be calculated by including in the continuous teaching content the same skeleton position information as in Figures 4 and 5. An example of calculation of the spatial deviation degree in the control unit 12 will be described with reference to Figures 6 to 9.

[0040] 6A to 6C are diagrams showing an example of calculating the spatial deviation degree from consecutive learning content, focusing on the point with the largest position change within a predetermined time in the first embodiment.

[0041] 6A is a diagram in which the cumulative value of the spatial movement amount of the feature point with the largest position change in the continuous teaching content within a predetermined time is plotted in units of time smaller than the predetermined time. For example, the diagram is divided into predetermined time periods of 0 to 8Δt and plotted with circles. If the point with the largest position change in the continuous teaching content within the predetermined time period is the right wrist 306, the spatial movement amount of the right wrist 306 is added for each Δt and plotted with a circle. The control unit 12 calculates the cumulative movement amount 60, which is the movement amount of the feature point of the continuous teaching content.

[0042] FIG. 6B is a diagram in which the amount of movement of the continuous teaching content is plotted as circles, the amount of spatial movement of the worker's right wrist 306 while imitating the movements of the continuous teaching content and working is added up for each Δt and plotted as triangles.

[0043] The control unit 12 compares the cumulative movement amount of the worker with the cumulative movement amount of the continuous teaching content at each divided time (nΔt lapse, n is an integer from 1 to 8). At Δt lapse, 2Δt lapse, and 3Δt lapse, the time difference at which the cumulative movement amount is the same is less than Δt, so the control unit 12 determines that the worker is following the movement of the continuous teaching content. At 4Δt lapse, the time difference A at which the cumulative movement amount is the same is more than Δt, so the control unit 12 determines that the worker is not following the movement of the continuous teaching content, and controls the presentation speed of the continuous teaching content to be reduced at 4Δt, as shown in FIG. 6C. At 5Δt lapse, the time difference B at which the cumulative movement amount is the same is more than Δt, so the control unit 12 determines that the worker is not following the movement of the continuous teaching content, and controls the presentation speed of the continuous teaching content to be further reduced at 5Δt, as shown in FIG. 6C. Similarly, when 6Δt has elapsed, the time difference that results in the same cumulative movement amount is less than Δt, so it is determined that the worker is following the action of the continuous teaching content, and the presentation speed of the continuous teaching content is controlled to the standard speed at 6Δt, as shown in Fig. 6C. Note that, since the presentation speed of the continuous teaching content is changed after 4Δt and 5Δt have elapsed, the cumulative movement amount of the continuous teaching content becomes cumulative movement amount 61 that is delayed from cumulative movement amount 60 of the standard continuous teaching content, and it is advisable to calculate the spatial deviation for each Δt for cumulative movement amount 61 after the content presentation speed is changed.

[0044] In the examples shown in Figures 6A to 6C, the description focuses on one point with the largest position change within a predetermined time, but position changes of multiple points may be focused on. For example, in the case of an action using both hands, the positions of the fingers of both hands may be set as feature points, and the spatial deviation may be calculated using multiple feature points. In addition, the spatial deviation may be calculated using statistical values ​​of the positions of multiple feature points (for example, the sum of coordinates).

[0045] In addition, the method of changing the presentation speed may be controlled stepwise using multiple thresholds. For example, if the cumulative movement amount at any time is r, the average speed of the continuous teaching content when the cumulative movement amount is r is v1, the elapsed time of the continuous teaching content is t1, the average speed of the worker when the cumulative movement amount is r is v2, and the elapsed time of the worker is t2, the time difference t2-t1 that results in the same cumulative movement amount can be calculated by the following formula. t2-t1=r(1 / v2-1 / v1) …(1)

[0046] Fig. 6D is a diagram showing another example of calculating the spatial deviation from the continuous teaching content by focusing on the point where the position change is greatest within a predetermined time in Example 1. In the examples shown in Figs. 6A to 6C, whether the worker is following the movement of the continuous teaching content is determined based on the time difference that results in the same cumulative movement amount, but in the example shown in Fig. 6D, whether the worker is following the movement of the continuous teaching content is determined based on the difference in the cumulative movement amount at the same time.

[0047] FIG. 6D shows the cumulative spatial movement amount of the feature point with the largest position change in the continuous teaching content plotted as a circle in time units (Δt) finer than a predetermined time, and plotted as a triangle when the worker imitates the movements of the continuous teaching content and adds up the spatial movement amount of the worker's right wrist 306 during the work for every Δt.

[0048] The control unit 12 compares the cumulative movement amount of the worker with the cumulative movement amount of the continuous teaching content at each divided time (after nΔt has elapsed, n being an integer from 1 to 8). At the time Δt has elapsed, 2Δt has elapsed, and 3Δt has elapsed, the difference between the cumulative movement amount is equal to or less than a predetermined threshold, so the control unit 12 determines that the worker is following the movement of the continuous teaching content. At the time 4Δt has elapsed, the difference between the cumulative movement amount of the worker and the cumulative movement amount of the continuous teaching content is equal to or more than a predetermined threshold, so the control unit 12 determines that the worker is not following the movement of the continuous teaching content, and controls the presentation speed of the continuous teaching content to be reduced at 4Δt, as shown in FIG. 6C. At the time 5Δt has elapsed, the difference between the cumulative movement amount of the worker and the cumulative movement amount of the continuous teaching content is equal to or more than a predetermined threshold, so the control unit 12 determines that the worker is not following the movement of the continuous teaching content, and controls the presentation speed of the continuous teaching content to be further reduced at 5Δt, as shown in FIG. 6C. Similarly, when 6Δt has elapsed, since the difference between the cumulative movement amount of the worker and the cumulative movement amount of the continuous teaching content is equal to or less than a predetermined threshold, it is determined that the worker is following the action of the continuous teaching content, and the presentation speed of the continuous teaching content is controlled to be the standard speed at 6Δt, as shown in Fig. 6C. Note that, since the presentation speed of the continuous teaching content is changed after 4Δt and 5Δt have elapsed, the cumulative movement amount of the continuous teaching content becomes cumulative movement amount 61 which is delayed from cumulative movement amount 60 of the standard continuous teaching content, and it is preferable to calculate the spatial deviation degree for every Δt for cumulative movement amount 61 after the content presentation speed is changed.

[0049] In the example shown in FIG. 6D, the description focuses on one point with the largest position change within a predetermined time, but any point with a position change of a predetermined amount or more (a point with movement of a predetermined distance or more) may be used. Also, attention may be paid to position changes of multiple points. For example, in the case of an action using both hands, the spatial deviation may be calculated using multiple feature points with the finger positions of both hands as feature points. Also, the spatial deviation may be calculated using a statistical value of the positions of multiple feature points (for example, the sum of coordinates).

[0050] In addition, it is preferable to control the change in the presentation speed in a stepwise manner. For example, in the above example, when the difference between the cumulative movement amount of the worker and the cumulative movement amount of the continuous teaching content exceeds a predetermined threshold value multiple times in succession, the presentation speed is gradually decreased.

[0051] In this way, the presentation speed of the continuous teaching content is controlled in multiple stages depending on the time difference or difference in the cumulative movement amount that results in the same cumulative movement amount, i.e., the degree of spatial dissociation, so that the worker can suitably follow the continuous teaching content.

[0052] Figures 7A to 7C are figures showing an example of calculating the spatial deviation between a worker and the continuous teaching content in Example 1 by focusing on the positions of feature points of the continuous teaching content and the position coordinates of the work information of the worker corresponding to the feature points.

[0053] FIG. 7A is a diagram showing the distance between the feature points of the worker and the feature points of the continuous teaching content. For example, when the skeletal information of the tip of the right index finger is used as the feature point, the diagram shows the time series change in the spatial distance between the position of the tip of the right index finger of the model in the continuous teaching content and the tip of the right index finger of the worker. In other words, if the worker correctly imitates the movements of the continuous teaching content, the distance between the feature points will be close to 0. FIG. 7B is a diagram showing an example of the presentation speed.

[0054] The operation of the control unit 12 will be described with reference to FIG. 7A and FIG. 7B. From time 0 to t1, the distance between the feature points is equal to or greater than a predetermined threshold, and the control unit 12 presents the content by setting the presentation speed of the continuous teaching content to 0. That is, the continuous teaching content is stopped, and the avatar in the continuous teaching content is presented to the worker in a stopped state. When the worker approaches the avatar in the continuous teaching content to imitate the movement, the distance between the feature points becomes equal to or less than the predetermined threshold after time t1. At this time, the control unit 12 presents the content at a presentation speed A times the standard speed, as shown in FIG. 7B. Next, the content is presented at A times the standard speed until time t4 when the distance between the feature points exceeds the predetermined threshold. When the distance between the feature points exceeds the predetermined threshold at time t4, the control unit 12 presents the content by setting the presentation speed of the continuous teaching content to 0. In this way, by controlling the playback speed of the content according to the distance between the worker and the avatar, the content is played when the worker approaches to imitate the continuous instructional content, and the content stops if the worker interrupts the work or performs an action different from the work that he or she should be performing, preventing the content from being missed.

[0055] In the example shown in FIG. 7B, the presentation speed of the continuous teaching content is controlled at two speeds, 0 and A times the standard speed, but other control methods may be used. For example, as shown in FIG. 7C, the playback speed of the continuous content may be changed continuously. In FIG. 7C, the presentation speed is increased as the distance between the feature points decreases from time t1, and the content is presented at a presentation speed B times the standard speed between time t2 and time t3, when the distance between the feature points is almost 0 (within the error range of FIG. 7A). Between time t3 and time t4, the presentation speed is decreased as the distance between the feature points increases. In this way, the playback speed of the content is controlled according to the distance between the worker and the avatar, thereby improving the follow-up ability of the worker to the continuous teaching content. Furthermore, by setting the value of B to a value of 1 or more, the work speed can be increased when follow-up is sufficient, thereby improving work efficiency.

[0056] FIG. 8 is a diagram for explaining whether or not the feature points of the continuous learning content according to the first embodiment are within the visual field of the worker.

[0057] For simplification, a case where the worker faces straight on the +z axis will be described. When the field of view (FOV) 80 of the MR device 5 is obtained, the spatial range that can be displayed on the MR device 5 is determined. For example, when the position coordinates (x1, y1, z1) of a feature point 81 of a continuous teaching content are known, a field of view range 83 in the xy plane at z=z1 that can be displayed on the MR device 5 is obtained. In the example shown in FIG. 8, the feature point 81 is included in the field of view range 83 of the worker and can be seen by the worker. On the other hand, the position coordinates (x2, y2, z1) of a feature point 82 of a different continuous teaching content are not included in the field of view range 83 of the worker and therefore cannot be seen by the worker. The process of the control unit 12 when the spatial deviation between the feature point of the continuous teaching content and the field of view range is thus determined will be described with reference to FIG. 9.

[0058] 9A to 9C are diagrams for explaining an example of the relationship between the feature points and the field of view of the continuous learning content in the first embodiment.

[0059] For simplicity, we will explain the case where the worker faces straight along the +z axis, as in Fig. 8. We also consider that the feature point in the continuous teaching content moves at a constant speed in the +x direction on the xy plane at z=z1.

[0060] FIG. 9A shows the time series change of the x coordinate of the feature point of the continuous teaching content when the presentation speed is not changed. In the case shown in FIG. 9A, x=xa shows the coordinate of the end of the -x side of the xy plane 83 at z=z1 in FIG. 8, and x=xb shows the coordinate of the end of the +x side. That is, the visual field range of the worker in the x direction is from xa to xb. Note that, when the content moves in the z-axis direction, the xa and xb thresholds that determine the visual field range change. For example, when the content moves in the z-axis + direction, xb-xa that determines the visual field range becomes larger, and when the content moves in the z-axis - direction, xb-xa that determines the visual field range becomes smaller. The feature point shown in FIG. 9A is outside the visual field of the worker at time t1, and the worker cannot see it.

[0061] The operation of the control unit 12 will be described with reference to FIG. 9A and FIG. 9B. The control unit 12 determines whether the feature point of the continuous instructional content is included in the visual field of the MR device 5. For example, between time 0 and t1 in FIG. 9A, the feature point of the continuous instructional content is included in the visual field, and the control unit 12 presents the continuous instructional content at a standard speed. If the control unit 12 determines at time t1 that the feature point of the continuous instructional content is not included in the visual field, the control unit 12 controls the presentation speed to gradually slow down. At time t2, the presentation speed of the content becomes 0, and the content is stopped. Furthermore, the control unit 12 determines whether the feature point of the continuous instructional content is included in the visual field, and if it determines that the feature point of the continuous instructional content is included in the visual field, the control unit 12 sets the presentation speed to a value equal to or less than 0 and plays the continuous instructional content in reverse. In FIG. 9B, the presentation speed is set to a negative value between time t2 and t3, so that the content is played in reverse. If the feature point of the continuous instructional content returns to the visual field at time t3, the presentation speed is set to 0, and the content is stopped. That is, by determining whether the feature point of the continuous instructional content is included in the visual field and changing the presentation speed, the feature point of the continuous instructional content moves within the visual field as shown in FIG. 9C. If the feature point of the continuous instructional content is significantly out of the visual field range and the reverse playback time exceeds a predetermined time, it is preferable not to reverse play the continuous instructional content. In this way, by controlling the playback speed of the content depending on whether the feature point of the continuous instructional content is included in the visual field, the content that has fallen outside the visual field of the MR device 5 can be brought back into the visual field, preventing the content from being overlooked.

[0062] FIG. 10 is a flowchart of the process executed by the control device 8 of the first embodiment.

[0063] First, the work information acquisition unit 10 acquires work information from the imaging apparatus 1 and the MR device 5, and sends it to the control unit 12 (S11).

[0064] Next, the control unit 12 acquires the continuous instruction content from the content database 14, and analyzes the acquired work information by referring to the acquired continuous instruction content, and calculates the spatial deviation degree (S12). For example, the spatial deviation degree is calculated based on at least one of the difference between the spatial movement amount of the feature point with the largest position change within a predetermined time and the spatial movement amount of the feature point of the continuous instruction content shown in FIG. 6B and FIG. 6D, the distance between the position of the feature point of the continuous instruction content and the position of the work information of the worker corresponding to the feature point shown in FIG. 7A, and the relationship between the feature point of the continuous instruction content and the field of view of the worker shown in FIG. 9A to FIG. 9C. Note that the above-mentioned examples may be combined to make a judgment. For example, the judgment may be made based on both the presence of the feature point of the continuous instruction content within the field of view of the MR device 5 and the distance from the feature point.

[0065] Next, the control unit 12 determines whether to control the content display method based on the calculated spatial dissociation degree (S13). In step S13, if the spatial dissociation degree is large, the worker is not following the continuous teaching content, so the control unit 12 determines to control the content display method and controls the content presentation speed in a direction to decrease (S14). Also, if the spatial dissociation degree is small, the worker is following the continuous teaching content, so the control unit 12 may determine to control the content display method and controls the content presentation speed in a direction to increase. In this way, by controlling the content display method based on the spatial dissociation degree, the worker's ability to follow the continuous teaching content can be improved, and work efficiency can be improved.

[0066] On the other hand, if it is determined in step S13 that the content display method is not to be controlled, the content presentation speed is maintained.

[0067] Then, the output unit 13 transmits the content to the MR device 5 at the presentation speed determined in step S14 or the current presentation speed, and presents the content whose presentation speed has been adjusted to the operator (S15).

[0068] As described above, according to the first embodiment of the present invention, the work efficiency of a worker can be improved by acquiring work information of the worker, detecting deviations from the ideal actions and states shown in the continuous teaching content, and dynamically controlling the display method of the content.

[0069] In this embodiment, a work support system using augmented reality is exemplified, but the present invention may be applied to a work support system using virtual reality. That is, the control device 8 may reconstruct the situation at the site into a virtual three-dimensional space based on three-dimensional information collected from one or more edge processing devices 2, acquire work information of the worker in the virtual three-dimensional space, detect deviations from ideal actions and states shown in the continuous teaching content, and dynamically control the display method of the content. The present invention may also be applied to a work support education system that improves the effectiveness of education for workers.

[0070] <Example 2> In the second embodiment, the deviation between the ideal motion or state and the motion or state of the worker is detected, and the information is added to the content to improve the work efficiency of the worker. In the second embodiment, the differences from the first embodiment described above are mainly described, and the description of the same configuration and functions as the first embodiment is omitted.

[0071] FIG. 11 is a flowchart of the process executed by the control device 8 of the second embodiment.

[0072] In the second embodiment, the task information acquisition unit 10 acquires task information from the imaging device 1 and the MR device 5, and sends the task information to the control unit 12 (S11). Next, the control unit 12 analyzes the acquired task information, calculates a spatial disparity (S12), and determines whether to control the content display method based on the calculated spatial disparity (S13). The control unit 12 controls the content presentation speed to be slowed down according to the determination result of step S13 (S14). The processes of steps S11 to S14 are the same as those of the first embodiment described above.

[0073] Thereafter, the control unit 12 adds information to the continuous teaching content or creates additional information to be presented to the worker (S20), thereby making it possible to make the worker aware of the presence of a spatial discrepancy.

[0074] For example, the color, size, and brightness of the continuous teaching content are changed according to the degree of spatial dissociation, and the continuous teaching content is made to blink, thereby making the worker aware that the degree of spatial dissociation is large. Specifically, when the degree of spatial dissociation is small, the color or border of the continuous teaching content is displayed in a cool color (such as blue), and when the degree of spatial dissociation is large, the color or border of the continuous teaching content is displayed in a warm color (such as red), thereby making it possible to present content that allows the worker to intuitively grasp the degree of spatial dissociation. Furthermore, when the degree of spatial dissociation is small, the continuous teaching content is displayed in a small size, and when the degree of spatial dissociation is large, the continuous teaching content is displayed in a large size, thereby making it possible to present content that is easy to view when the degree of spatial dissociation is large. Furthermore, when the degree of spatial dissociation is large, the continuous teaching content is displayed in a bright size, thereby making it possible to present content that is easy to view when the degree of spatial dissociation is large. In addition, by slowly flashing the continuous teaching content when the spatial deviation is small, and constantly displaying it when the spatial deviation is large, the worker can visually confirm only the action of the worker at the timing when the continuous teaching content disappears while imitating the continuous teaching content, and can confirm whether the continuous teaching content is being imitated correctly. As another method, the trajectory of the continuous teaching content may be displayed to allow the worker to grasp the spatial deviation.

[0075] Further, as a method other than adding information to the continuous instruction content, a text may be displayed to make the worker aware of the difference in the operation, an image of the correct work procedure may be displayed, an alarm sound may be output, or tactile information such as vibration may be provided to inform the worker that the spatial deviation is large. For example, as shown in the example of FIG. 9A, when the continuous instruction content is outside the field of view of the MR device 5, information indicating the position where the continuous instruction content is stopped may be presented as additional information. Specifically, the head position information of the avatar in the continuous instruction content may be referenced, and a difference value between the head position information of the worker may be displayed to allow the worker to grasp the position of the continuous instruction content. Also, a straight line connecting a feature point of the continuous instruction content and a corresponding feature point of the worker may be displayed on the MR device 5 so that the position of the continuous instruction content can be intuitively understood.

[0076] Then, the output unit 13 transmits the content to which the information has been added in step S20 to the MR device 5, and presents the content, the presentation speed of which has been adjusted, to the worker (S15).

[0077] In this way, the worker can easily grasp the degree of deviation from the ideal operation or state, thereby improving the worker's work efficiency.

[0078] As described above, according to the second embodiment of the present invention, in addition to the effects of the first embodiment, by adding information to the content, the worker can be made aware of deviations from ideal actions or states, thereby improving the work efficiency of the worker.

[0079] <Example 3> In the third embodiment, the motion determining unit 200 determines whether the motion of the worker is correct, and if the motion of the worker is not correct, motion correction content is created to ensure that the worker performs the correct work. In the third embodiment, differences from the first embodiment described above will be mainly described, and the description of the same configuration and functions as the first embodiment will be omitted.

[0080] FIG. 12 is a logical block diagram of the control device 8 according to the third embodiment.

[0081] The control device 8 of the third embodiment has a work information acquisition unit 10, a calculation unit 11, a communication unit 15, an operation determination unit 200, and a corrective content creation unit 201. The calculation unit 11 has a control unit 12, an output unit 13, and a content database 14. The work information acquisition unit 10, the calculation unit 11, the control unit 12, the output unit 13, the content database 14, and the communication unit 15 are the same as those of the first embodiment described above.

[0082] The action determination unit 200 determines whether the worker is performing a correct action based on the work information of the worker. For example, when the feature point of the continuous teaching content needs to pass through a predetermined point (x, y, z), if the worker performs an action without the feature point of the worker passing through a predetermined range from this point, the action determination unit 200 determines that the worker has performed an incorrect action.

[0083] When the action determination unit 200 determines that the action is incorrect, the corrective content creation unit 201 creates corrective content for returning the incorrect action to a correct action. For example, when a feature point should pass through a predetermined point (x, y, z) but work information is observed in which the feature point passed through a different point (x', y', z'), the corrective content creation unit 201 creates corrective content for moving the feature point from the position coordinates where the current worker information was observed to the predetermined point (x, y, z). When presenting the corrective content, a text display or an icon such as a figure that informs the worker that the previous action was incorrect can be displayed, allowing the worker to easily recognize that the action was incorrect.

[0084] FIG. 13 is a flowchart of the process executed by the control device 8 of the third embodiment.

[0085] In the third embodiment, the work information acquisition unit 10 acquires work information from the imaging apparatus 1 and the MR device 5, and sends the work information to the control unit 12 (S11). Next, the control unit 12 analyzes the acquired work information and calculates the spatial disparity (S12).

[0086] Next, the motion judging unit 200 analyzes whether the worker is performing a correct motion (S30). When the motion judging unit 200 judges that the motion is incorrect, the corrective content creating unit 201 creates corrective content (S31) and starts presenting the created corrective content (S32). Then, the created corrective content executes the loop of steps S11 to S15 of the second embodiment (FIG. 11) until the presentation of the corrective content ends, and presents the corrective content with added information to the worker at a controlled presentation speed (S33, S34). In this way, corrective content can be generated to return to the correct motion when an incorrect motion is performed, and can be presented to the worker, ensuring that the correct work is performed.

[0087] On the other hand, if the action determination unit 200 determines that the action is correct, the control unit 12 determines whether to control the content display method based on the calculated spatial disparity (S13). The control unit 12 controls the content presentation speed to slow down according to the determination result of step S13 (S14). The processes of steps S11 to S14 are the same as those of the first embodiment described above.

[0088] As described above, according to the third embodiment of the present invention, in addition to the effects of the first embodiment, it is possible to reduce operational mistakes.

[0089] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0090] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0091] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0092] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0093] 1. Imaging device 2 Edge processing equipment 3. Device to be operated 4. Network 5. Mixed Reality Devices 6. Internet 7. Cloud 8 Control device 10 Work information acquisition unit 11 Arithmetic section 12 Control section 13 Output section 14 Content Database 15 Communications Department 40 Continuous Instructional Content 41 Hand Skeleton for Continuous Teaching Content 42 Worker's fingers 43 Hand skeleton of a worker 60 Cumulative movement of continuous instructional content 61 Cumulative movement amount after changing content presentation speed 61 70 Threshold of distance between feature points 80 MR Device 5 FOV80 81 Characteristics of Continuous Instructional Content 82 Characteristics of Continuous Instructional Content 83 Range of the xy plane at z=z1 101 Processor 102 Memory 103 Auxiliary storage device 104 Communication Interface 105 Input Interface 106 Output Interface 107 Keyboard 108 Mouse 109 Display device 200 Operation judgment section 201 Corrective Content Creation Department 300 Head position information 301 Neck position information 302 Left shoulder position information 302 Left shoulder position information 303 Right shoulder position information 304 Waist position information 305 Right elbow position information 306 Right wrist position information

Claims

1. A control device that uses cross reality to present three-dimensional movements to a worker, The computer includes a computing device that executes a predetermined computation process and a storage device that is accessible by the computing device, The arithmetic device includes an acquisition unit that acquires work information of a work being performed by the worker; a calculation unit configured to perform a calculation for presenting continuous teaching content corresponding to a task being performed by the worker to the worker; the computing device includes an output unit that outputs content, A control device characterized in that the calculation unit has a control unit that dynamically controls a display method of the continuous teaching content depending on the degree of spatial deviation between the work information acquired by the acquisition unit and the continuous teaching content.

2. The control device according to claim 1 , The control device is characterized in that the control unit changes a presentation speed of the continuous teaching content according to a degree of spatial deviation between the task information and the continuous teaching content.

3. The control device according to claim 2, A control device characterized in that the control unit slows down the playback speed of the continuous teaching content or stops playback of the continuous teaching content when the spatial deviation between the work information and the continuous teaching content is greater than a predetermined threshold.

4. The control device according to claim 2, The control device is characterized in that, when a spatial deviation between the task information and the continuous teaching content is greater than a predetermined threshold, the control unit plays the continuous teaching content in reverse.

5. The control device according to claim 2, The control unit is characterized in that if the times at which a first feature point, among the feature points included in the work information, that has the largest position change within a specified time period and a second feature point included in the continuous teaching content that corresponds to the first feature point, exhibit the same amount of spatial movement are separated by a specified threshold or more, the control unit determines that the spatial deviation between the work information and the continuous teaching content is large.

6. The control device according to claim 2, The control unit is characterized in that, when the distance between the spatial movement amount of a first feature point that has the largest position change within a specified time among the feature points included in the work information and the spatial movement amount of a second feature point included in the continuous teaching content that corresponds to the first feature point at the same time is greater than a specified threshold, the control unit determines that the spatial deviation between the work information and the continuous teaching content is large.

7. The control device according to claim 2, The control device is characterized in that the control unit controls the presentation speed of the continuous instructional content in a plurality of stages depending on the degree of spatial separation.

8. The control device according to claim 2, The control unit is characterized in that, when a distance between a first feature point of the worker's body included in the work information and a second feature point of the avatar's body included in the continuous teaching content corresponding to the first feature point is greater than a predetermined threshold, the control unit determines that there is a large degree of spatial discrepancy between the work information and the continuous teaching content.

9. The control device according to claim 2, The control device is characterized in that, when the worker is not viewing the continuous teaching content, the control unit determines that a degree of spatial deviation between the work information and the continuous teaching content is large.

10. The control device according to claim 1 , The control device is characterized in that the control unit adds additional information created in accordance with the task information acquired by the acquisition unit to the continuous learning content.

11. The control device according to claim 10, A control device characterized in that the additional information is one or more of the color of the continuous instructional content, the size of the continuous instructional content, the brightness of the continuous instructional content, the blinking of the continuous instructional content, the trajectory of points in the continuous instructional content, text information provided along with the continuous instructional content, work procedure images provided along with the continuous instructional content, sound provided along with the continuous instructional content, and tactile information provided along with the continuous instructional content.

12. The control device according to claim 10, The control device is characterized in that the calculation unit creates additional information to enable the continuous teaching content to be viewed when the worker is not viewing the continuous teaching content, and adds the created additional information to the continuous teaching content.

13. The control device according to claim 1 , The control device, wherein the calculation unit controls to display motion correction content for the worker to return to a correct motion, in accordance with the work information acquired by the acquisition unit.

14. An information presentation method in which a control device presents a three-dimensional action to a worker by using cross reality, comprising: the control device is configured by a computer having an arithmetic unit that executes a predetermined arithmetic process and a storage device that can be accessed by the arithmetic unit; The information presentation method includes: an acquisition step in which the arithmetic device acquires work information of the work being performed by the worker; A calculation procedure in which the calculation device executes a calculation for presenting continuous teaching content corresponding to a task being performed by the worker to the worker; an output step in which the arithmetic device outputs the content; The information presentation method is characterized in that the calculation procedure includes a control procedure that dynamically controls a display method of the continuous teaching content depending on the degree of spatial deviation between the work information acquired in the acquisition procedure and the continuous teaching content.

Images