Task support system and task support method

Abstract

To provide a task support system and a task support method that can support a task performed by a plurality of workers.SOLUTION: A task support system is used to support a task for a plurality of fastening locations of an article present in a real space. The task support system includes a first display device and a second display device, which can display virtual objects to overlap the real space. Each of the plurality of fastening locations is assigned to any of a plurality of groups. The first display device displays a first virtual object for a first fastening location included in a first group, which is one of the plurality of groups. The second display device displays a second virtual object for a second fastening location included in a second group, which is another one of the plurality of groups.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to a work assistance system and a work assistance method. [Background technology]

[0002] Conventionally, display devices that can be worn by workers have been used to work efficiently. Display devices can display virtual space overlaid on real space and provide various information to workers. Workers can perform their work more efficiently by referring to the information displayed by the display device. There is a demand for technology that can support work by multiple workers using display devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-156237 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a work support system and a work support method that can support work performed by a plurality of workers. [Means for solving the problem]

[0005] A work support system according to an embodiment is used to support work on multiple fastening points of an item present in real space. The work support system includes a first display device and a second display device capable of displaying virtual objects superimposed on the real space. Each of the multiple fastening points is assigned to one of multiple groups. The first display device displays a first virtual object for a first fastening point included in a first group, which is one of the multiple groups. The second display device displays a second virtual object for a second fastening point included in a second group, which is another of the multiple groups. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a work support system according to an embodiment. [Figure 2] FIG. 2 is a schematic view illustrating a display device according to the embodiment. [Figure 3] FIG. 3 is a schematic diagram illustrating the state of work. [Figure 4] FIG. 4 is a plan view showing the article to be worked on. [Figure 5] FIG. 5 is a schematic diagram illustrating the state of the work. [Figure 6] FIG. 6 is a schematic diagram showing an example of output by the display device according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram illustrating master data to be referenced in the embodiment of the present invention. [Figure 8] FIG. 8 is a table illustrating an example of fastening point master data. [Figure 9] FIG. 9 is a plan view showing the article to be worked on. [Figure 10] 10(a) and 10(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 11] 11(a) and 11(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 12]12(a) and 12(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 13] 13(a) and 13(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 14] FIG. 14 is a schematic diagram showing a specific example of a virtual object. [Figure 15] 15(a) and 15(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 16] 16(a) and 16(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 17] 17(a) and 17(b) are schematic diagrams illustrating the state of work. [Figure 18] FIG. 18 is a schematic diagram showing an example of a tool. [Figure 19] FIG. 19 is a schematic diagram showing the state of the work. [Figure 20] FIG. 20 is a schematic diagram for explaining the calculation method in the first embodiment of the present invention. [Figure 21] 21(a) and 21(b) are schematic diagrams for explaining the calculation method in the first embodiment of the present invention. [Figure 22] FIG. 22 is a schematic diagram for explaining the calculation method in the first embodiment of the present invention. [Figure 23] FIG. 23 is a schematic diagram for explaining the calculation method in the first embodiment of the present invention. [Figure 24] FIG. 24 is a schematic diagram illustrating the state of work. [Figure 25] Figure 25(a) is a bottom view showing the article to be worked on, and Figure 25(b) is a plan view showing the article to be worked on. [Figure 26] 26(a) and 26(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 27] 27(a) and 27(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 28] 28(a) and 28(b) are schematic diagrams showing examples of output by the display device according to the embodiment. [Figure 29] FIG. 29 is a flowchart showing a task assistance method according to the embodiment. [Figure 30] FIG. 30 is a schematic diagram showing the configuration of another task support system according to the embodiment. [Figure 31] 31(a) and 31(b) are schematic diagrams showing the configuration of another task support system according to the embodiment. [Figure 32] FIG. 32 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those already described are designated by the same reference numerals, and detailed description will be omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a work support system according to an embodiment. A task support system 1 according to the embodiment is used to support a task. As shown in Fig. 1, the task support system 1 includes a display device 100a (first display device) and a display device 100b (second display device). In the illustrated example, the task support system 1 further includes a terminal device 100c, a terminal device 100d, and a host computer 100e.

[0009] The display device 100a is worn by a worker who performs work. The display device 100a is connected to a terminal device 100c via wireless communication. Data related to the work is communicated between the display device 100a and the terminal device 100c. The display device 100b is worn by another worker who performs work. The display device 100b is connected to a terminal device 100d via wireless communication. Data related to the work is communicated between the display device 100b and the terminal device 100d. The terminal devices 100c and 100d are connected to a host computer 100e via wired communication, wireless communication, or a network. The host computer 100e communicates data related to the work with each of the terminal devices 100c and 100d.

[0010] In the example shown in FIG. 1, the work support system 1 includes two display devices. The work support system 1 may include three or more display devices. Here, an example in which the work support system 1 includes two display devices will be described. In the following description, when there is no particular distinction between the display device 100a and the display device 100b, at least one of the display device 100a and the display device 100b will be simply referred to as the "display device 100."

[0011] FIG. 2 is a schematic view illustrating a display device according to the embodiment. The display device 100 includes, for example, a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170, as shown in FIG. 2 .

[0012] In the illustrated example, the display device 100 is a twin-eye head-mounted display. Two lenses, 111 and 112, are fitted into a frame 101. Projection devices 121 and 122 project information onto the lenses 111 and 112, respectively.

[0013] The projection devices 121 and 122 display the recognition results of the worker's body, virtual objects, etc. on the lenses 111 and 112. Only one of the projection devices 121 and 122 may be provided, and information may be displayed on only one of the lenses 111 and 112.

[0014] The lenses 111 and 112 are optically transparent. A person wearing the display device 100 can view the real world through the lenses 111 and 112. The person wearing the display device 100 can also view information projected onto the lenses 111 and 112 by the projection devices 121 and 122. Information (virtual space) is displayed superimposed on the real space through projection by the projection devices 121 and 122.

[0015] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 emits infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor that can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts audio input.

[0016] The processing device 150 controls each element of the display device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in accordance with the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0017] The battery 160 supplies the power required for operation to each element of the display device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the display device 100 and may communicate with the processing device 150.

[0018] The display device 100 is not limited to the illustrated example, and may be a single-lens head-mounted display. The display device may be in the form of glasses as illustrated, or in the form of a helmet.

[0019] A worker uses a tool to tighten or loosen a screw on an object. Hereinafter, tightening a screw and loosening a screw will be referred to as turning a screw. The object may be a part, unit, or semi-finished product for manufacturing a product. The tool may be a wrench, a screwdriver, or the like. Here, an example in which an embodiment of the present invention is applied to a fastening operation for tightening a screw will be mainly described.

[0020] 3 and 5 are schematic diagrams illustrating the state of work, and Fig. 4 is a plan view showing the article to be worked on. 3, workers W1 and W2 are working on an item 200. Worker W1 wears a display device 100a and turns a screw on the item 200. Worker W2 wears a display device 100b and turns a screw on the item 200.

[0021] , On the top surface of the article 200, there are fastening points (screw holes) into which screws are turned. For example, as shown in FIG. 3, cylindrical members 210, 220, and 230 are arranged on the top of the article 200. As shown in FIG. 4, around the periphery of the member 210, there are fastening points 211-214 (an example of first fastening points) for fixing the member 210. Similarly, around the periphery of the member 220, there are fastening points 221-224 (an example of second fastening points). Around the periphery of the member 230, there are fastening points 231-234 (an example of third fastening points). Workers W1 and W2 use wrenches and extension bars to turn the screws into each fastening point in turn.

[0022] As shown in FIG. 5, a marker 205 is provided near the work object. The marker 205 is an AR marker. As will be described later, the marker 205 is provided to set the base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 205. Alternatively, instead of a marker, the base point may be indicated by a hand gesture. The processing device 150 sets the three-dimensional coordinate system based on multiple points indicated by the hand gesture.

[0023] At the start of the work, each display device 100 sets a three-dimensional coordinate system. Specifically, the image camera 131 and depth camera 132 of each display device 100 capture an image of the marker 205. The processing device 150 recognizes the marker 205 from the captured image. The processing device 150 sets the three-dimensional coordinate system based on the position and orientation of the marker 205.

[0024] FIG. 6 is a schematic diagram showing an example of output by the display device according to the embodiment. During work, the image camera 131 and the depth camera 132 capture images of the item 200, the worker's left hand 261, and the worker's right hand 262. The processing device 150 recognizes the left hand 261 and the right hand 262 from the captured images by hand tracking. The processing device 150 may cause the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the display device or the processing device displaying information on the lens using the projection device will also be simply referred to as "displaying information."

[0025] 6, the processing device 150 displays the recognition results of the left hand 261 and the right hand 262 superimposed on the hands in real space. In the example shown, a plurality of virtual objects 261a and a plurality of virtual objects 262a are displayed as the recognition results of the left hand 261 and the right hand 262. The plurality of virtual objects 261a represent the plurality of joints of the left hand 261, respectively. The plurality of virtual objects 262a represent the plurality of joints of the right hand 262, respectively. Instead of the joints, virtual objects (meshes) representing the surface shapes of the left hand 261 and the right hand 262, respectively, may be displayed.

[0026] When the processing device 150 recognizes the left hand 261 and the right hand 262, it measures the position of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may also be used as the position of the hand. Alternatively, the position of the center of gravity of the entire hand may also be used as the position of the hand.

[0027] The processing device 150 further calculates the position and orientation of the display device 100. As an example, the processing device 150 calculates the position and orientation of the display device 100 using a spatial mapping function. More specifically, the depth camera 132 measures the distance between the display device 100 and surrounding objects. Surface information of the surrounding objects is obtained from the measurement results (depth image) by the depth camera 132. The surface information includes the position and orientation of the object's surface. For example, the surface of each object is represented by multiple meshes, and the position and orientation are calculated for each mesh. The processing device 150 calculates the relative position and orientation of the display device 100 with respect to the surfaces of the surrounding objects from the surface information.

[0028] Once the markers 205 are recognized, the position of each surface is also expressed in a three-dimensional coordinate system with the markers 205 as the base point. The position and orientation of the display device 100 in the three-dimensional coordinate system are calculated from the positional relationship between the surface of the object and the display device 100. Here, the orientation of the display device 100 refers to the direction in front of the display device 100. For example, when a worker wears the display device 100, the orientation of the display device 100 is parallel to the front direction of the worker's face.

[0029] Spatial mapping is repeatedly performed at predetermined intervals. Each time spatial mapping is performed, surface information of surrounding objects is obtained. The processing device 150 calculates changes in the position and orientation of the surface between the latest spatial mapping result and the previous spatial mapping result. In a situation where surrounding objects are not moving, changes in the position and orientation of the surface correspond to changes in the position and orientation of the display device 100, respectively. The processing device 150 calculates the amount of change in the position and orientation of the display device 100 from the changes in the position and orientation of the surface. The detection results of the sensor 140 may also be used to calculate the amount of change in the position and orientation of the display device 100. The processing device 150 updates the position and orientation of the display device 100 based on the obtained amount of change. Note that, instead of spatial mapping, an existing positioning method may be used to acquire the position and orientation of the display device 100.

[0030] FIG. 7 is a schematic diagram illustrating master data to be referenced in the embodiment of the present invention. When a task is performed, various data are referenced. For example, as shown in Fig. 7, task master data 300, base point master data 310, fastening point master data 320, and tool master data 330 are referenced.

[0031] The job ID, job name, item ID, and item name are registered in the job master data 300. The job to be performed is specified by the job ID, job name, the ID of the item to be worked on, or the name of the item.

[0032] A method for setting a base point for each task is registered in the base point master data 310. The processor 150 acquires the method for setting a base point for the selected task, and sets the base point according to that setting method.

[0033] The fastening point master data 320 stores, for each operation, the identification information (ID) of the fastening point where the screw is turned, the position of the fastening point, the order of the operation, etc. Details of the fastening point master data 320 will be described later.

[0034] The tool master data 330 registers the ID, tool model, tool length, socket model, socket length, etc. of the tool to be used for each task. The tool model indicates the classification of the tool based on structure, shape, performance, etc. The tool length is the length from the center of rotation to the grip when the tool is used to tighten a screw. The socket model indicates the classification of the socket based on structure or shape. The socket length indicates the length of the socket in the direction connecting the tool and the screw when the screw is tightened. If an extension bar is used, the model and length of the extension bar are also registered in the tool master data 330.

[0035] FIG. 8 is a table illustrating an example of fastening point master data. The fastening point master data 320 shown in FIG. 8 includes identification information 321 of the item (product to be produced), identification information 322 of the work, category 323, group 324, identification information 325 of the fastening point, position 326, sequence 327, number of times 328, and torque value 329.

[0036] Identification information 321 is identification information unique to each item. Identification information 322 is identification information unique to each operation. Category 323 indicates the group to which each fastening point belongs. Category 323 is determined based on the position of each fastening point on the item or the relationship between the structure of the item and each fastening point. For example, multiple fastening points that are close to each other on the item are grouped together as one category. Alternatively, multiple fastening points corresponding to one part, one unit, or one semi-finished product are grouped together as one category. In the example shown in FIGS. 3 and 4 , fastening points 211-214, fastening points 221-224, and fastening points 231-234 correspond to member 210, member 220, and member 230, respectively. Therefore, the group of fastening points 211-214, the group of fastening points 221-224, and the group of fastening points 231-234 are each treated as different categories.

[0037] Group 324 indicates which group each fastening point belongs to. The method of assigning a group to each fastening point is arbitrary. For example, if data for category 323 exists, multiple fastening points that belong to the same category are grouped together. If data for category 323 does not exist, multiple fastening points that are close to each other are grouped together.

[0038] Identification information 325 is identification information for each fastening point. Unique identification information is expressed for each fastening point by a combination of identification information 321, category 323, and identification information 325. Position 326 indicates the position of each fastening point. The position is expressed in a three-dimensional coordinate system set using marker 205. Order 327 indicates the order of work for each fastening point. When category 323 is set for each fastening point, the order of work is set for each category.

[0039] The number of times 328 indicates the number of times a screw is tightened at each fastening point. If the number of times set in the number of times 328 is 2 or more, the number set in the order 327 indicates the order in the first screw tightening. In the order 327, the screw tightening order may be set for each number of screw tightenings. The torque value 329 indicates the torque value required to tighten a screw at each fastening point.

[0040] The fastening points 211 to 214, the fastening points 221 to 224, and the fastening points 231 to 234 are each assigned to one of a plurality of groups in the fastening point master data 320.

[0041] For example, various master data are prepared in advance before work and saved in a storage area of ​​the host computer 100e. The host computer 100e may generate some of the data included in the fastening point master data 320. For example, if data for category 323 exists, the host computer 100e assigns one group to multiple fastening points that belong to the same category and generates data for group 324. If data for category 323 and order 327 does not exist, the host computer 100e assigns one group to multiple fastening points that are close to each other based on data for position 326.

[0042] The host computer 100e may generate data for the group 324 based on the respective positions of the display device 100a and the display device 100b. Specifically, the host computer 100e acquires the respective positions of the display device 100a and the display device 100b via the terminal devices 100c and 100d. The host computer 100e calculates the distance between each of the display device 100a and the display device 100b and each fastening point. The host computer 100e assigns one group to a predetermined number of fastening points, starting with those closest to the display device 100a and the display device 100b.

[0043] FIG. 9 is a plan view showing the article to be worked on. In the illustrated example, the group of fastening points 211-214, the group of fastening points 221-224, and the group of fastening points 231-234 are each treated as a different section. The order of work is predetermined for each section. Therefore, as shown in Fig. 9, the fastening points 211-214, the fastening points 221-224, and the fastening points 231-234 are assigned to a first group G1, a second group G2, and a third group G3, respectively.

[0044] When work is performed, the host computer 100e acquires the fastening location master data 320. The host computer 100e assigns, for each group, a worker (display device) to perform the work at the fastening location. For example, the host computer 100e acquires the respective positions of the display device 100a and the display device 100b via the terminal devices 100c and 100d. The host computer 100e calculates the distance between each of the display device 100a and the display device 100b and the fastening locations belonging to each group. The host computer 100e assigns the group to which the fastening location closest to the display device 100a belongs to the display device 100a. The host computer 100e assigns the group to which the fastening location closest to the display device 100b belongs to the display device 100b.

[0045] The host computer 100e transmits the fastening location master data 320 and the allocation of workers for each group to the terminal devices 100c and 100d. Based on the received data, the terminal device 100c determines the fastening location where the wearer of the display device 100a will be working. The display device 100a transmits the determination result to the display device 100a. Based on the received data, the terminal device 100d determines the fastening location where the wearer of the display device 100b will be working. The terminal device 100d transmits the determination result to the display device 100b.

[0046] Figures 10(a), 10(b), 11(a), 11(b), 12(a), 12(b), 13(a), and 13(b) are schematic diagrams showing examples of output by a display device according to an embodiment. When the display device 100a and the display device 100b receive the data indicating the fastening location where the work is to be performed, they display a virtual object at the fastening location.

[0047] As an example, the display device 100a (worker W1) is assigned a first group G1, and the display device 100b (worker W2) is assigned a second group G2. It is determined that the wearer of the display device 100a will perform work at the fastening location 211. As shown in FIG. 10(a), the display device 100a displays a virtual object 411 (an example of a first virtual object) at the fastening location 211. It is determined that the wearer of the display device 100b will perform work at the fastening location 221. As shown in FIG. 10(b), the display device 100b displays a virtual object 421 (an example of a second virtual object) at the fastening location 221.

[0048] The virtual object 411 is displayed so as not to overlap with the fastening point 211, and is located near the fastening point 211. For example, the distance between the fastening point 211 and the virtual object 411 is shorter than the distance between other fastening points and the virtual object 411. Similarly, the virtual object 421 is displayed so as not to overlap with the fastening point 221, and is located near the fastening point 221. The distance between the fastening point 221 and the virtual object 421 is shorter than the distance between other fastening points and the virtual object 421.

[0049] Worker W1 follows the display of virtual object 411 and performs work on fastening location 211 where virtual object 411 is displayed. Worker W2 follows the display of virtual object 421 and performs work on fastening location 221 where virtual object 421 is displayed. By displaying virtual objects 411 and 421, workers can easily understand the fastening location where they need to work.

[0050] As shown in FIGS. 10(a) and 10(b), the display device 100a and the display device 100b may display messages 451 and 452, respectively, in addition to the virtual objects. The message 451 indicates work instructions for the worker W1. The message 452 indicates work instructions for the worker W2. For example, if it is necessary to adjust the timing of work by the worker W1 and the timing of work by the worker W2, the display of the messages 451 and 452 allows the workers W1 and W2 to easily understand whether it is okay to perform the work. The work instructions may be output as a video as shown in the figures, or as an audio output.

[0051] When the work on fastening point 211 is completed, terminal device 100c refers to sequence 327 of fastening point master data 320 and determines the fastening point to be worked on next. Terminal device 100c transmits the determination result to display device 100a. As shown in FIG. 11(a), display device 100a displays a virtual object 413 (another first virtual object) at fastening point 213 (another first fastening point). Similarly, when the work on fastening point 221 is completed, terminal device 100d determines the fastening point to be worked on next. Terminal device 100d transmits the determination result to display device 100b. As shown in FIG. 11(b), display device 100b displays a virtual object 423 (another second virtual object) at fastening point 223 (another second fastening point).

[0052] During the work, the terminal devices 100c and 100d create a work record of which fastening points the screws have been turned into. Based on the work record, it is possible to determine when work on each fastening point in the first group G1 and work on each fastening point in the second group G2 have been completed. When the work assigned to worker W1 or the work assigned to worker W2 is completed, the host computer 100e assigns work on each fastening point in the third group G3 to the worker who finished their work earlier.

[0053] As an example, worker W1 finishes the task earlier than worker W2. In this case, the host computer 100e assigns tasks to each fastening point of the third group G3 to the display device 100a. As shown in FIG. 12(a), the display device 100a displays a virtual object 431 (an example of a third virtual object) at the fastening point 231 (third fastening point) of the third group G3.

[0054] As shown in FIG. 12(a), the display device 100a may display a message 453. The message 453 indicates a work instruction to worker W1. After worker W2 finishes the work, worker W2 waits until worker W1 finishes the work. At that time, as shown in FIG. 12(b), the display device 100b may display a message 454. The message 454 indicates a wait instruction to worker W2.

[0055] When worker W1 finishes the work, the first screw tightening is completed for all fastening points. After that, the second screw tightening is performed for each fastening point. If the order of the second screw tightening is not determined, the display device 100a and the display device 100b display virtual objects at all fastening points, as shown in, for example, FIGS. 13(a) and 13(b). In the illustrated example, virtual objects 411 to 414 are displayed at fastening points 211 to 214, respectively. Virtual objects 421 to 424 are displayed at fastening points 221 to 224, respectively. Virtual objects 431 to 434 are displayed at fastening points 231 to 234, respectively. As illustrated, a message 455 and a message 456 may also be displayed.

[0056] Worker W1 and worker W2 sequentially perform work on the fastening locations where the virtual objects are displayed. The display of the virtual objects may disappear for the fastening locations where work has been performed. Alternatively, the display of the virtual objects may change. This allows each worker to easily understand which fastening locations have not been screwed a second time.

[0057] Alternatively, similar to the examples of FIGS. 10(a) to 11(b), one virtual object may be sequentially displayed at each fastening location. For example, the display device 100a calculates the distance between the display device 100a and each fastening location. The display device 100a displays a virtual object at the closest fastening location where the second work has not been performed. Similarly, the display device 100b calculates the distance between the display device 100b and each fastening location. The display device 100b displays a virtual object at the closest fastening location where the second work has not been performed.

[0058] When the order of the second screw tightening is determined, the host computer 100e assigns each group to the display device 100a and the display device 100b, as in the first screw tightening. The terminal device 100c and the terminal device 100d determine the fastening locations to be worked on. The display device 100a and the display device 100b display the virtual objects in the determined order.

[0059] FIG. 14 is a schematic diagram showing a specific example of a virtual object. The virtual object may include information about the task. For example, as shown in Fig. 13, the virtual object 411 includes task information such as identification information 411a, a specified torque value 411b, a detection value 411c, a meter 411d, a ratio 411e, and a count 411f. The identification information 411a is unique identification information assigned to the fastening point 211 and is expressed as a character string. The specified torque value 411b is a torque value required to fasten the screw into the fastening point 211 and is specified in advance.

[0060] A tool capable of detecting torque values ​​may be used in the work. In that case, detected value 411c indicates the torque value detected by the tool. Meter 411d indicates the specified torque value and the detected torque value. Ratio 411e indicates the ratio of the detected value to the specified torque value. Some work requires fastening a screw multiple times into one fastening point. In that case, count 411f indicates the number of times the screw has been fastened into fastening point 211.

[0061] Virtual objects 412 to 414, virtual objects 421 to 424, and virtual objects 431 to 434 also include information about the work to be done at each fastening location, similar to the example shown in Fig. 14. The worker performs the work while checking the content displayed in the virtual objects. Displaying the information necessary for the work in the virtual objects can improve the efficiency of the work.

[0062] 15(a), 15(b), 16(a), and 16(b) are schematic diagrams showing examples of output by the display device according to the embodiment. The display device 100 may display virtual objects in a form different from the virtual objects shown in FIGS. 10(a) to 13(b). For example, as shown in FIG. 15(a), the display device 100a may display a spherical virtual object 411x at the fastening point 211. The virtual object 411x is displayed so as to be superimposed on the fastening point 211. Similarly, as shown in FIG. 15(b), the display device 100b may display a spherical virtual object 421x at the fastening point 221. The virtual object 421x is displayed so as to be superimposed on the fastening point 221. The virtual object 411x is another example of a first virtual object. The virtual object 421x is another example of a second virtual object.

[0063] The virtual objects 411x and 421x are displayed at the fastening locations where work is about to be performed. By displaying the virtual objects 411x and 421x, the workers W1 and W2 can easily grasp the fastening locations where work is about to be performed.

[0064] 16(a), the display device 100a may display virtual objects 411y and 411z at the fastening point 211. The display device 100b may display virtual objects 421y and 421z at the fastening point 221. The virtual object 411y is spaced apart from the fastening point 211 in the direction of the screw holes in the fastening point 211. The virtual object 411z is located between the fastening point 211 and the virtual object 411y. The virtual object 421y is spaced apart from the fastening point 221 in the direction of the screw holes in the fastening point 221. The virtual object 421z is located between the fastening point 221 and the virtual object 421y. The virtual objects 411y and 411z are yet another example of a first virtual object. The virtual objects 421y and 421z are yet another example of a second virtual object.

[0065] Virtual objects 411y, 411z, 421y, and 421z are suitable for use when a wrench and an extension bar are used. Virtual object 411y indicates the position where a hand should be placed when turning a screw into fastening point 211. Virtual object 411z indicates the position where an extension bar should be placed when turning a screw into fastening point 211. For example, the distance between fastening point 221 and virtual object 411y corresponds to the length of the extension bar.

[0066] The worker W1 positions the extension bar so that the extension bar is close to or in contact with the virtual object 411z. The worker W1 also grips the head of the wrench so that his hand comes into contact with the virtual object 411y. By displaying the virtual objects 411y and 411z, the worker W1 can easily understand the positions of the tool and his hand when turning a screw into the fastening point 211. Similarly, by displaying the virtual objects 421y and 421z, the worker W2 can easily understand the positions of the tool and his hand when turning a screw into the fastening point 221. This can improve work efficiency.

[0067] In the illustrated example, virtual objects 411x, 411y, 421x, and 421y are spherical. Virtual objects 411z and 421z are rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each virtual object is not limited to this example. For example, virtual objects 411x, 411y, 421x, and 421y may be cubes. Virtual objects 411z and 421z may be linear.

[0068] After the virtual object is displayed, the processing device 150 may determine whether a predetermined object has come into contact with the virtual object. For example, the processing device 150 of the display device 100a determines whether the wearer's hand has come into contact with the virtual object 411x or 411y. Specifically, the processing device 150 calculates the distance between the position of the hand and the position of the virtual object. If the distance is less than a preset threshold, the processing device 150 determines that the hand has come into contact with the virtual object.

[0069] 15(a) to 16(b), the diameters of spherical virtual objects 411x, 411y, 421x, and 421y correspond to the threshold values. The spheres indicate the range within which it is determined that the hand has come into contact with the virtual object.

[0070] 17(a) and 17(b) are schematic diagrams illustrating the state of work. When the wrench 251 is used, the hand comes into contact with the virtual object 411x as shown in Fig. 17(a). When the wrench 251 and the extension bar 252 are used, the hand comes into contact with the virtual object 411y as shown in Fig. 17(b). In these cases, the processing device 150 determines that the hand has come into contact with the virtual object.

[0071] FIG. 18 is a schematic diagram showing an example of a tool. The processing device 150 may determine whether the tool has come into contact with a virtual object. For example, as shown in FIG. 18 , a wrench 251 has multiple markers 251a attached thereto. The processing device 150 recognizes the multiple markers 251a from an image captured by the image camera 131. The processing device 150 measures the position of each marker 251a. The positional relationship between the multiple markers 251a and the head 251b of the wrench 251 is registered in advance. The processing device 150 calculates the position of the head 251b based on the positions of at least three recognized markers 251a and the previously registered positional relationship. The processing device 150 calculates the distance between the position of the head 251b and the position of the virtual object. If the distance is less than a preset threshold, the processing device 150 determines that the wrench 251 has come into contact with the virtual object.

[0072] When a predetermined object comes into contact with a virtual object, it can be estimated that a screw is being turned into the fastening point 211 corresponding to the virtual object. In the example shown in Fig. 17(a), since a hand or a tool has come into contact with virtual object 411x, it can be estimated that a screw is being turned into the fastening point 211 corresponding to virtual object 411x. In the example shown in Fig. 17(b), since a hand or a tool has come into contact with virtual object 411y, it can be estimated that a screw is being turned into the fastening point 211 corresponding to virtual object 411y.

[0073] In addition to contact between real objects and virtual objects, the fastening location being worked on may also be estimated using hand movements. While a screw is being turned with a tool, the hand turning the tool moves in an arc. At this time, the position of the center of rotation remains almost constant. For example, while a screw is being turned with a wrench, the position of the wrench head remains almost constant. When the change in the position of the center of rotation becomes small, it can be estimated that the screw is being turned.

[0074] FIG. 19 is a schematic diagram showing the state of the work. For example, as shown in FIG. 19 , a worker uses a wrench 251 to tighten a screw at a fastening point. Here, an example will be described in which an extension bar 252 is not used. The worker places a screw 254 in a screw hole at a fastening point (not shown). The worker holds the grip of the wrench 251 with his right hand and fits the tip (head) of the wrench 251, to which a socket is attached, into the screw 254. The worker rotates the wrench 251 to turn the screw 254.

[0075] While the worker turns the wrench 251, the processing device 150 repeatedly measures the position of the hand. At this time, the hand is positioned on a circumference with a part of the wrench 251 as its center. The hand is moved in an arc. The processing device 150 uses this to estimate the position of the center of rotation of the tool. Based on the position of the center of rotation, the processing device 150 estimates the fastening point where the screw is being turned, the execution of work on the fastening point, etc. For example, the following first or second calculation method is used to calculate the center position.

[0076] 20, 21(a), 21(b), 22, and 23 are schematic diagrams for explaining the calculation method in the first embodiment of the present invention. In the first calculation method, the processing device 150 extracts three mutually different positions from the multiple measured positions. The processing device 150 calculates the circumcenter O of the three positions. Here, as shown in FIG. 22 , the three positions are defined as P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3), respectively. The position of the circumcenter O is defined as P0 (x0, y0, z0). Furthermore, for a triangle obtained by connecting positions P1 to P3, the length of the opposite side of position P1 is defined as L1. The length of the opposite side of position P2 is defined as L2. The length of the opposite side of position P3 is defined as L3. The angle at position P1 is defined as α. The angle at position P2 is defined as β. The angle at position P3 is defined as γ. In this case, the position of the circumcenter O is expressed by the following mathematical formula (1). Note that in mathematical formula (1), symbols with arrows indicate position vectors. Equation (1) can be rewritten as equation (2). Equation (2) can be decomposed into equations (3) to (5).

number

number

number

number

number

[0077] From the formulas (3) to (5), x0, y0, and z0 are calculated. The processing device 150 calculates the position P0 (x0, y0, z0) of the circumcenter O as the center position of the rotation of the wrench 251.

[0078] The center position of the rotation of the wrench 251 can be considered to be the position at which the screw 254 is turned by the wrench 251. It can then be estimated that the screw is being tightened at the fastening point closest to the center position. For example, the processing device 150 extracts combinations of three positions from multiple hand positions measured over a predetermined time interval and calculates the center position. The processing device 150 repeats the extraction of position combinations and the calculation of the center position. The processing device 150 calculates the distance between each center position calculated over that time interval and the fastening point, and if any of the distances is less than a threshold, it estimates that the screw is being tightened at that fastening point.

[0079] When a digital tool capable of detecting torque values ​​is used, the detection results by the tool may be used to estimate the operation. For example, if the distances between each center position and the fastening point within a predetermined time span are all less than a threshold value and a torque value is detected by the tool, the processing device 150 estimates that a screw is being turned into the fastening point.

[0080] In the first calculation method described above, the length of a tool interposed between the wrench 251 and the screw 254 may be used in the calculation to more accurately estimate the position of the screw. In the example shown in FIG. 19 , a socket 253 is fitted into the wrench 251. That is, the center position of rotation of the wrench 251 and the position of the screw 254 are separated by the length of the socket 253. If the length of the socket 253 is registered in advance, the processing device 150 can more accurately estimate the position of the screw 254 using the center position and the length of the socket 253.

[0081] When estimating the position of the screw 254 using the length of the socket 253, it is necessary to determine on which side the screw 254 is located with respect to the plane on which the wrench 251 is rotating. In the example shown in FIG. 21(a), the wrench 251 is rotating in a rotation direction RD1. The screw 254 and the socket 253 are located on the lower side. In the example shown in FIG. 21(b), the wrench 251 is rotating in a rotation direction RD2. The rotation direction RD2 is opposite to the rotation direction RD1. The screw 254 and the socket 253 are located on the upper side with respect to the plane parallel to the rotation direction RD2.

[0082] To determine the side on which the screw 254 is located, the processing device 150 uses the center position, two positions of the hand, time-series information on the two positions, and screw loosening information. For example, as shown in FIG. 22, the two positions are P1 (x1, y1, z1) and P2 (x2, y2, z2), respectively. The center position is P0 (x0, y0, z0). The time when the hand was at position P1 and the time when the hand was at position P2 are known. That is, the processing device 150 holds time-series information on positions P1 and P2. In this example, the time when the hand was at position P1 was before the time when the hand was at position P2.

[0083] The loosening / reducing information indicates whether the screw is being tightened or loosened. If the wrench 251 is a digital tool, the wrench 251 determines whether the screw is being tightened or loosened from the detected torque value and generates the loosening / reducing information. The processing device 150 may determine whether the screw is being tightened or loosened from the time-series data of the torque value received from the wrench 251 and generate the loosening / reducing information.

[0084] The plane passing through the positions P0 to P2 is expressed by the following formula (6): In formula (6), k, l, m, and n are constants. [Number 6] kx+ly+mz+n=0

[0085] By substituting the positions P0 to P2 into formula (6), the following formulas (7) to (9) are obtained: From formulas (7) to (9), the constants k, l, m, and n are calculated. [Number 7] kx0+ly0+mz0+n=0 [Number 8] kx1+ly1+mz1+n=0 [Number 9] kx2+ly2+mz2+n=0

[0086] Here, the processing device 150 calculates a vector from the center position P0 to a position P1 at an earlier time. The processing device 150 also calculates a vector from the center position P0 to a position P2 at a later time. The screw 254 is located at a position P0 on the normal vector P0P1×P0P2, which is a distance L0 from the center position P0. Q When the screw is tightened and the time at position P1 is earlier than the time at position P2, the processing unit 150 calculates a normal vector P0P1×P0P2 between vector P0P1 and vector P0P2.

[0087] From position P0, position P where the wrench and socket act on the screw Q The length to is expressed by the following formula (10): In the formulas below, an arrow attached to a symbol indicates that the value indicated by the symbol is a vector.

number

[0088] On the other hand, from position P0 to position P Q The vector up to can also be expressed by the following equation (11): In equation (11), t is a constant.

number

[0089] By substituting Equation (11) into Equation (10), the following Equation (12) is obtained. The length L0 in Equation (12) is registered in advance. By solving Equation (12), t is calculated.

number

[0090] Once t is calculated, the position P is calculated using the position P0, constants k, l, m, n, and t. Q is calculated, i.e., the position of the screw is obtained.

[0091] For example, the processing device 150 extracts a combination of three positions from the multiple positions of the screw 254 calculated over a predetermined time interval, and calculates the center position. The processing device 150 repeats the extraction of the combinations of positions and the calculation of the center position. If the distance between each position of the screw 254 and the fastening point over that time interval is less than a threshold, the processing device 150 estimates that the screw is being turned into that fastening point.

[0092] As shown in FIG. 17(b), there is a case where the screw is fastened via an extension bar 252. In this case, the position of the screw 254 can be estimated using the length of the extension bar 252, as in the method described above. That is, the screw 254 is located on the normal vector P0P1×P0P2 at a distance from the center position P0 that is the sum of the length of the extension bar 252 and the length of the socket 253. The position P0 of the screw 254 can be calculated using the center position P0, the length of the extension bar 252, and the length of the socket 253. Q By taking into consideration the length of another tool interposed between the screw 254 and the wrench 251, the position of the screw 254 can be estimated with higher accuracy.

[0093] In the second calculation method, the center position of rotation for each fastening point is registered in advance. For example, as shown in FIG. 23, center positions c1 to c4 are registered in advance for fastening points 211 to 214. Three positions p1 to p3 of the hand are calculated from an image of the hand turning the screw. The processing device 150 calculates distances d1 to d3 between the center position c1 and positions p1 to p3, respectively. Similarly, the processing device 150 calculates the distance between each of the center positions c2 to c4 and positions p1 to p3, respectively. The processing device 150 calculates the variation in the distance from each of the center positions to positions p1 to p3. If any of the variations is less than a threshold, the processing device 150 estimates that the tool is rotating at that center position. The processing device 150 estimates that the screw is being turned at the fastening point associated with the estimated center position.

[0094] The variation may be the sum of the differences between the average value of the multiple distances and each distance, the variance of the multiple distances, or the standard deviation of the multiple distances. In the example shown in Fig. 23, hand positions p1 to p3 are approximately equidistant from the center position c1. Therefore, it is estimated that the screw is being turned to the fastening point 211 associated with the center position c1.

[0095] When a digital tool is used, the detection results of the tool may be used to estimate the operation. For example, if the center position is estimated and the torque value is detected by the tool, the processing device 150 estimates that a screw is being turned into the fastening point associated with the center position.

[0096] The processing device 150 may repeat the first or second calculation method regardless of whether an operation is being performed. Specifically, the processing device 150 executes the first or second calculation method using multiple hand positions obtained over a predetermined time span. If the execution of an operation is not estimated from the multiple hand positions over that time span, the processing device 150 slides the time span and executes the first or second calculation method again. As an example, the time span is set to 6 seconds, and the slide amount is set to 16 milliseconds. The time span and slide amount are set appropriately depending on the performance of the processing device 150.

[0097] When no work is being performed, the tool is not actually rotating and there is no center of rotation. However, it is possible to calculate the apparent center position from multiple positions of the hand. While no work is being performed, the calculated center position is far from the position of the fastening point. Therefore, it is not estimated that work is being performed. When work is being performed, the calculated center position approaches the position of the fastening point. The processing device 150 estimates the start of work as the time when it is first estimated that work is being performed.

[0098] For example, when the display device 100a and the display device 100b estimate work to be performed on any of the fastening locations, they transmit the estimation result to the terminal device 100c or the terminal device 100d. Based on the estimation result, the terminal device 100c or the terminal device 100d may link a work record to the data of the estimated fastening location. The work record indicates that a screw has been turned into the fastening location. This automatically creates a work record.

[0099] When a digital tool such as a digital torque wrench or digital torque driver is used, the processing device 150 receives the detected torque value from the tool. The torque value required for fastening may be set in advance, and the digital tool may determine whether the required torque value has been detected. The digital tool transmits the determination result to the processing device 150. The digital tool also transmits the rotation angle, the time when the torque value was detected, and the like to the processing device 150.

[0100] The display devices 100a and 100b transmit the data received from the digital tool to the terminal devices 100c and 100d, respectively. The terminal devices 100c and 100d associate the received maximum torque value or judgment result with the data of the work location. This automatically creates a more detailed work record.

[0101] The processing device 150 may determine whether work on the estimated fastening location has been completed based on the received torque value. If the received torque value is equal to or greater than a preset torque value, the processing device 150 determines that work on the fastening location has been completed. When the processing device 150 receives a determination result indicating whether the required torque value has been detected from the tool, the processing device 150 may determine whether work on the fastening location has been completed based on the determination result.

[0102] When it is determined that the work has been completed, the display device 100a and the display device 100b transmit the determination result to the terminal device 100c or the terminal device 100d. Upon receiving the determination result, the terminal device 100c or the terminal device 100d associates the work record with the data of the estimated fastening location.

[0103] When a screw is tightened multiple times for one fastening point, the processing device 150 may count the number of times the screw is tightened. When it is estimated that the work is being performed and then determined that the work is completed, the number of times the screw is tightened is counted once.

[0104] The virtual object 411x, the virtual object 411y, or the virtual object 411z may be displayed in addition to the virtual object 411. When the virtual object 411x or the virtual object 411y is displayed, the virtual object 411 may be displayed after the work on the fastening location 211 is estimated. The virtual object 411 includes information about the work. Therefore, the size of the virtual object 411 is larger than the size of the virtual object 411x or the virtual object 411y. A large virtual object 411 is likely to overlap with the article 200. If the virtual object 411 overlaps with the article 200, it becomes difficult for the worker to visually recognize the article 200. By displaying the virtual object 411 after the start of the work, the display of the virtual object 411 is less likely to interfere with the work. This can further improve the convenience of the display device 100.

[0105] Fig. 24 is a schematic diagram illustrating the state of work. Fig. 25(a) is a bottom view showing the item to be worked on. Fig. 25(b) is a plan view showing the item to be worked on. Figs. 26(a), 26(b), 27(a), 27(b), 28(a), and 28(b) are schematic diagrams showing examples of output by the display device according to the embodiment. 3 and 4, work may be performed on an article 200a shown in Fig. 24. Workers W1 and W2 perform work on each fastening point of the article 200a. Cylindrical members 210 and 220 are placed on the article 200a.

[0106] Fastening points exist on the bottom and top surfaces of the article 200a. As shown in Fig. 25(a), fastening points 211-214 and fastening points 221-224 exist on the bottom surface of the article 200a. As shown in Fig. 25(b), fastening points 231-234 and fastening points 241-244 exist on the top surface of the article 200a.

[0107] The fastening points 231 to 234 are located around the member 210. The fastening points 211 to 214 are located directly below the fastening points 231 to 234, respectively. The fastening points 211 to 214 and 231 to 234 are provided to fix the member 210.

[0108] The fastening points 241 to 244 are located around the member 220. The fastening points 221 to 224 are located directly below the fastening points 241 to 244, respectively. The fastening points 221 to 224 and 241 to 244 are provided to fix the member 220.

[0109] When the work is performed, the host computer 100e acquires the fastening point master data. For example, as shown in Figures 25(a) and 25(b), the fastening points 211-214, the fastening points 221-224, the fastening points 231-234, and the fastening points 241-244 are assigned to a first group G1, a second group G2, a third group G3, and a fourth group G4, respectively.

[0110] The fastening location master data may include the order of work by group. As an example, the first group G1 and the second group G2 are to be worked on before the third group G3 and the fourth group G4. The first group G1 is assigned to the display device 100a (worker W1), and the second group G2 is assigned to the display device 100b (worker W2). It is determined that the worker W1 will perform work on the fastening location 211. It is determined that the worker W2 will perform work on the fastening location 221.

[0111] First, workers W1 and W2 get under the article 200a and, while looking up, perform work on the fastening points 211 and 221, respectively. At this time, the display device 100a displays a virtual object 411 at the fastening point 211, as shown in Fig. 26(a). The display device 100b displays a virtual object 421 at the fastening point 221, as shown in Fig. 26(b).

[0112] 26(a) and 26(b), the display devices 100a and 100b may display messages 461 and 462 for workers W1 and W2, respectively. The messages 461 and 462 indicate work instructions for workers W1 and W2, respectively.

[0113] For example, worker W1 finishes his work earlier than worker W2. In this case, the host computer 100e assigns work to each fastening location of the third group G3 to the display device 100a. However, the work for the third group G3 must be performed after the work for both the first group G1 and the second group G2 has been completed. Worker W1 must wait until worker W2 finishes his work. In this case, the display device 100a may display a message 463 as shown in FIG. 27(a). The message 463 instructs worker W1 to wait.

[0114] While worker W1 is waiting, worker W2 works on second group G2. Fig. 27(b) shows a display example when work on fastening points 221 to 223 has been completed and worker W2 is working on fastening point 224. A virtual object 424 is displayed at fastening point 224.

[0115] After that, when worker W2 finishes the work on the second group G2, worker W1 can start the work on the third group G3. For example, as shown in FIG. 28(a), a virtual object 431 is displayed at the joining point 231. Furthermore, worker W2 is assigned the work on the fourth group G4. As shown in FIG. 28(b), a virtual object 441 is displayed at the joining point 241.

[0116] 28(a), the message 463 changes to a message 464 indicating work instructions to the worker W1. The display device 100b also displays a message 465 indicating work instructions to the worker W2.

[0117] In the examples shown in FIGS. 26(a) to 28(b), instructions may be output by voice instead of or in addition to the messages.

[0118] FIG. 29 is a flowchart showing a task assistance method according to the embodiment. When the work support method shown in FIG. 29 is executed, the various master data shown in FIG. 7 are referenced as appropriate. First, a work to be performed is selected (step S1). For example, the terminal device 100c, the terminal device 100d, or the host computer 100e accepts the selection of the work. The work to be performed is selected by a worker or a work manager using a work ID or an item ID. The terminal device 100c, the terminal device 100d, or the host computer 100e may determine the work to be performed based on data obtained from the image camera 131 or other sensors. The terminal device 100c, the terminal device 100d, or the host computer 100e selects the work based on the determination result.

[0119] Next, the image camera 131 captures an image of the marker 205. The processing device 150 sets the origin of a three-dimensional coordinate system based on the position and orientation of the marker 205 (step S2).

[0120] The host computer 100e refers to the fastening location master data 320 and determines whether work should be performed in groups (step S3). For example, if the fastening location master data 320 sets the order of work at fastening locations for each category or group, it is determined that work in groups is necessary. If the fastening location master data 320 does not set the order of work, it is determined that work in groups is not necessary. The host computer 100e transmits the fastening location master data 320 and the result of the determination to the terminal devices 100c and 100d.

[0121] The terminal devices 100c and 100d determine the fastening points to be worked on based on the determination result in step S3 (step S4). When work is performed in groups, the fastening points to be worked on are determined according to the order within the group. When an order of work between groups is set, the fastening points are determined with reference to that order. When work is not performed in groups, the fastening point closest to the display device 100 is determined as the fastening point to be worked on.

[0122] The terminal devices 100c and 100d determine whether the determined fastening locations are workable (step S5). The terminal devices 100c and 100d transmit the determination results to the display devices 100a and 100b, respectively. For example, if the worker must wait until another group's work is completed, it is determined that the work is not workable. In this case, the processing device 150 outputs a wait instruction to the worker (step S6). The wait instruction may be displayed as a message or output as a voice message. Step S5 is then executed again, and it is again determined whether the work is workable.

[0123] If the work on the determined fastening location can be performed in parallel with work on another group, or if the work does not need to wait for the completion of work on the other group, the processing device 150 displays a virtual object at the fastening location and outputs work instructions to the worker (step S7). When the virtual object is displayed and when it is determined that the work has been performed, the processing device 150 references the fastening location master data 320 and the tool master data 330. The work instructions may be displayed as a message or output as audio. The processing device 150 determines whether the work has been performed on the fastening location determined in step S4. If it is determined that the work has been performed, the terminal device 100c or 100d creates a work record for the fastening location estimated to have been worked on (step S8).

[0124] The record created in step S8 is saved in the history data 340. For example, the torque value detected by the tool is linked to the ID of the work and the ID of the estimated fastening point. As shown in the figure, the processing device 150 may further link the type and ID of the tool used, the number of times the screw was tightened, and the mark recognition result to the ID of the fastening point. The mark is recognized by the processing device 150 from an image taken by the image camera 131. The processing device 150 extracts a cluster of pixels of the mark color from the image and counts the number of pixels in that cluster. If the number of pixels exceeds a preset threshold, it is determined that a mark has been attached.

[0125] After step S8, the processing device 150 determines whether all work has been completed (step S9). If all work has not been completed, step S3 is executed again. For example, if the fastening points included in one of the groups have been worked on in the immediately preceding step S4 and there are fastening points in that group that have not yet been worked on, the work on the fastening points that have not yet been worked on is determined in the next steps S3 and S4. In this way, work on a group basis is repeated as appropriate.

[0126] The advantages of the embodiment will be described. Multiple workers may work together on large items. Traditionally, each worker would communicate with each other about which fastening points they had worked on. However, checking each other's work takes time and reduces work efficiency. Furthermore, when workers are far apart or the work area is noisy, it can be difficult for workers to communicate with each other. As a result, there is a risk that work will be performed in the wrong order, a single fastening point will be worked on multiple times, or fastening points will not be worked on at all.

[0127] An embodiment of the present invention is used to support work by multiple workers. The work support system 1 includes multiple display devices 100. Each worker wears a display device 100 while working. The display device 100 can display virtual objects superimposed on real space. By displaying the virtual objects, the workers can grasp the fastening points to be worked on or can check work information from the virtual objects. This can improve work efficiency.

[0128] Furthermore, when work is performed, fastening location master data relating to a plurality of fastening locations is acquired. In the fastening location master data, each fastening location is assigned to one of a plurality of groups. When the fastening location master data is acquired, the display device 100a displays a virtual object 411 for the fastening location 211 included in the first group G1, as shown in FIG. 9(a), for example. The display device 100b displays a virtual object 421 for the fastening location 221 included in the second group G2, as shown in FIG. 9(b).

[0129] That is, each display device displays virtual objects for fastening points of different groups. Therefore, it is possible to prevent work from being unintentionally performed multiple times for one fastening point. Furthermore, by displaying virtual objects in a preset order, it is possible to prompt the worker to perform work in that order for each group. Therefore, it is possible to prevent work from being performed in the wrong order. It is also possible to reduce the possibility of fastening points not being worked on.

[0130] According to the embodiment of the present invention, it is possible to support work by a plurality of workers.

[0131] The display device 100 can estimate the work to be performed at each fastening point using the positions of the wearer's hands. For example, the display device 100a repeatedly measures the position of the hand of the worker W1. The display device 100a estimates the work to be performed at the fastening point 211 from the contact between the hand and a virtual object or the movement of the hand. The display device 100b repeatedly measures the position of the hand of the worker W2. The display device 100b estimates the work to be performed at the fastening point 221 from the contact between the hand and a virtual object or the movement of the hand.

[0132] Once the work is estimated, a work record can be automatically created indicating whether each fastening point has been worked on or not. This eliminates the need for workers to create work records. Work efficiency can be further improved. Furthermore, the work record can be used to distinguish between fastening points that have been worked on and fastening points that have not yet been worked on. Based on the results of this distinction and a preset order, fastening points that still need to be worked on can be determined, and virtual objects can be displayed at those fastening points.

[0133] When either worker W1 or W2 finishes the work for the assigned group, worker W1 or W2 is assigned the work for the next group. In the example shown in FIGS. 10(a) to 12(b), if worker W1 has finished the work for the first group G1 and worker W2 has not yet finished the work for the second group G2, worker W1 is assigned the work for the third group G3. As shown in FIG. 12(a), the display device 100a displays a virtual object 431 at the joining point 231 of the third group G3. By assigning the next work to a worker who works quickly, the entire work can be completed more quickly.

[0134] Depending on the task, one worker may have to wait for the other. For example, for the item 200a shown in FIG. 24, the work on fastening points 211-214 and 221-224 on the bottom surface must be completed before the work on fastening points 231-234 and 241-244 on the top surface can be completed. For example, worker W1 works on fastening points 211-214, and worker W2 works on fastening points 221-224. If worker W1 finishes his work before worker W2, worker W1 must wait for worker W2 to finish his work. If worker W1 turns a screw in a fastening point on the top surface of item 200a before worker W2 finishes his work, the work will have to be redone. In this case, the screw at the fastening point that was just worked on is returned to its original state, and the work is resumed after worker W2 finishes his work.

[0135] Regarding this problem, the display device 100 determines whether the fastening points can be worked on in parallel. In the above example, the worker W1 has finished working on the fastening points 211 to 214. The worker W1 has time to work on the fastening points 231 to 234. However, the work on the fastening points 231 to 234 needs to be performed after the work on the fastening points 221 to 224 is completed. Therefore, the display device 100a determines that the work on the fastening points 231 to 234 cannot be performed in parallel with the work on the fastening points 221 to 224. In this case, a wait instruction (message 463) is output as shown in FIG. 27(a). This can prompt the worker to perform the work more appropriately.

[0136] When the order of work on fastening points is not determined, the display device 100a and the display device 100b display a virtual object at the nearest fastening point that has not yet been worked on, thereby reducing the worker's effort to move around and improving work efficiency.

[0137] For example, after the first work has been performed at each fastening location, if the order of the second work has not been determined, the display device 100a and the display device 100b display a virtual object at the nearest fastening location where the second work has not been performed.

[0138] 30, 31(a), and 31(b) are schematic diagrams showing the configuration of another task support system according to the embodiment. The configuration of the task support system 1 according to the embodiment is not limited to the example shown in Fig. 1. As in the task support system 1a shown in Fig. 30(a), the host computer 100e may be omitted. In that case, the terminal device 100c or 100d has the function of the host computer 100e.

[0139] As in the task support system 1b shown in FIG. 31(a), the terminal device 100d and the host computer 100e may be omitted. In that case, the terminal device 100c has the functions of the terminal device 100d and the host computer 100e and is connected to both the display devices 100a and 100b. As in the task support system 1c shown in FIG. 31(b), the terminal device 100c, the terminal device 100d, and the host computer 100e may be omitted. In that case, the display devices 100a and 100b have the functions of the terminal devices 100c and 100d, respectively. One of the display devices 100a and 100b has the functions of the host computer 100e.

[0140] For example, the display device 100 is an AR device that displays augmented reality (AR) or an MR device that displays mixed reality (MR). When the display device 100 is realized as an MR device, it can detect contact between a virtual object and a human body. Therefore, as shown in FIGS. 17(a) and 17(b), when contact between a predetermined object and a virtual object is detected, it is preferable that the display device 100 is an MR device.

[0141] FIG. 32 is a schematic diagram showing a hardware configuration. 32 is used as the processing device 150 of the display device 100, the terminal device 100c, the terminal device 100d, and the host computer 100e. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0142] The ROM 92 stores a program that controls the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to perform each of the above-described processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0143] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0144] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) or the like.

[0145] An input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0146] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can output data to the output device via the output I / F 96.

[0147] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0148] The data processing by the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0149] The various data processing operations described above may be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, or DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0150] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0151] Embodiments of the invention include the following features. (Feature 1) A work support system for supporting work on a plurality of fastening points of an article present in real space, a first display device and a second display device capable of displaying a virtual object superimposed on the real space; Each of the plurality of fastening points is assigned to one of a plurality of groups, the first display device displays a first virtual object for a first fastening point included in a first group that is one of the plurality of groups; The second display device displays a second virtual object for a second fastening point included in a second group, which is another one of the plurality of groups. (Feature 2) The first display device uses a hand position of a wearer of the first display device to estimate an operation on the first fastening location; and The work support system according to Feature 1, wherein the second display device estimates the work to be performed on the second fastening location using the position of the hand of the wearer of the second display device. (Feature 3) The work support system described in Feature 2, wherein when work has been performed on all of the one or more first fastening locations included in the first group, the first display device displays a third virtual object for a third fastening location included in a third group, which is yet another one of the plurality of groups. (Feature 4) The work support system described in Feature 2, wherein the first display device outputs a standby instruction when work has been performed on all of the one or more first fastening locations included in the first group and work has not been performed on any of the one or more second fastening locations included in the second group. (Feature 5) The work support system according to any one of features 2 to 4, wherein when work has been performed on all of the plurality of fastening locations, the first display device and the second display device each display a virtual object for the closest fastening location. (Feature 6) the first virtual object includes one or more selected from a specified torque value required for fastening the screw to the first fastening point, a torque value detected in fastening the screw to the first fastening point, and a number of times the screw is fastened to the first fastening point, The work support system described in any one of features 1 to 5, wherein the second virtual object includes one or more selected from a specified torque value required to tighten the screw into the second fastening location, a torque value detected when tightening the screw into the second fastening location, and a number of times the screw is tightened into the second fastening location. (Feature 7) The work support system according to any one of Features 1 to 6, wherein the first display device and the second display device set a three-dimensional coordinate system in a virtual space using a marker provided in the real space as a base point. (Feature 8) A work assistance method for assisting work on a plurality of fastening points of an article present in real space, comprising: Each of the plurality of fastening points is assigned to one of a plurality of groups; a first display device capable of displaying a virtual object superimposed on the real space displays a first virtual object for a first fastening point included in a first group that is one of the plurality of groups; A work support method in which a second display device capable of displaying a virtual object superimposed on the real space displays a second virtual object for a second fastening location included in a second group, which is another one of the plurality of groups.

[0152] According to the embodiment described above, a work support system and a work support method capable of supporting work by a plurality of workers are provided.

[0153] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0154] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other. [Explanation of symbols]

[0155] 1, 1a to 1c: work support system, 100, 100a, 100b: display device, 100c, 100d: terminal device, 100e: host computer, 101: frame, 111, 112: lens, 121, 122: projection device, 131: image camera, 132: depth camera, 140: sensor, 141: microphone, 150: processing device, 160: battery, 170: storage device, 200, 200a: article, 205: marker, 210, 220, 230: component, 211 to 214, 221 to 224, 231 to 234, 241 to 244: fastening point, 251: wrench, 251a: marker, 251b: head, 252: extension bar, 253: Socket, 254: Screw, 261: Left hand, 261a: Virtual object, 262: Right hand, 262a: Virtual object, 300: Work master data, 310: Base point master data, 320: Fastening point master data, 330: Tool master data, 340: History data, 411: Virtual object, 411a: Identification information, 411b: Specified torque value, 411c: Detected value, 411d: Meter, 411e: Ratio, 411f: Number of times, 411x~411z, 412~414, 421, 421x~421z, 422~424, 431~434, 441: Virtual object, 451~456, 461~465: Message, G1: First group, G2: Second group, G3: 3rd group, G4: 4th group, W1, W2: workers

Claims

1. A work support system for supporting work on a plurality of fastening points of an article present in real space, a first display device and a second display device capable of displaying a virtual object superimposed on the real space; Each of the plurality of fastening points is assigned to one of a plurality of groups, the first display device displays a first virtual object for a first fastening point included in a first group that is one of the plurality of groups; The second display device displays a second virtual object for a second fastening point included in a second group, which is another one of the plurality of groups.

2. The first display device estimates an operation on the first fastening location using a hand position of a wearer of the first display device; The work assistance system according to claim 1 , wherein the second display device estimates the work to be performed on the second fastening location using a position of a hand of the wearer of the second display device.

3. 3. The work support system according to claim 2, wherein when work has been performed on all of the one or more first fastening locations included in the first group, the first display device displays a third virtual object for a third fastening location included in a third group that is yet another one of the plurality of groups.

4. 3. The work support system of claim 2, wherein the first display device outputs a standby instruction when work has been performed on all of the one or more first fastening locations included in the first group and work has not been performed on any of the one or more second fastening locations included in the second group.

5. The work support system according to claim 2 , wherein when work has been performed on all of the plurality of fastening locations, the first display device and the second display device each display a virtual object for the closest fastening location.

6. the first virtual object includes one or more selected from a specified torque value required for fastening the screw to the first fastening point, a torque value detected in fastening the screw to the first fastening point, and a number of times the screw is fastened to the first fastening point, The work support system according to any one of claims 1 to 5, wherein the second virtual object includes one or more selected from a specified torque value required to tighten the screw into the second fastening point, a torque value detected when tightening the screw into the second fastening point, and a number of times the screw is tightened into the second fastening point.

7. 6. The work support system according to claim 1, wherein the first display device and the second display device set a three-dimensional coordinate system in a virtual space using a marker provided in the real space as a base point.

8. A work assistance method for assisting work on a plurality of fastening points of an article present in real space, comprising: Each of the plurality of fastening points is assigned to one of a plurality of groups, a first display device capable of displaying a virtual object superimposed on the real space displays a first virtual object for a first fastening point included in a first group that is one of the plurality of groups; A work support method in which a second display device capable of displaying a virtual object superimposed on the real space displays a second virtual object for a second fastening location included in a second group, which is another one of the plurality of groups.

Images