Display device, acquisition system, processing method, program, and storage medium

Abstract

To provide a display device, an acquisition system, a processing method, a program, and a storage medium, which can perform a task more safely.SOLUTION: A display device displays a virtual object to overlap a real space. The display device acquires a position and a direction of the display device. The display device displays a first virtual object indicating a permissible task range set using the position and the direction. The display device sets a task position at a predetermined position for a fastening location of an article present in the real space. The display device determines whether or not the task position is within the permissible task range, and outputs a first alert when the task position is outside the permissible task range.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a display device, an acquisition system, a processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, display devices capable of displaying a virtual space overlaid on a real space have been used to improve work efficiency. Display devices can provide various information to workers. By referring to the displayed information, workers can perform their work more efficiently. However, there is a demand for technology that allows workers to perform work more safely using display devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-156237 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a display device, an acquisition system, a processing method, a program, and a storage medium that enable work to be performed more safely. [Means for solving the problem]

[0005] A display device according to an embodiment displays a virtual object superimposed on a real space. The display device acquires the position and orientation of the display device. The display device displays a first virtual object indicating a workable range set using the position and orientation. The display device sets a work position at a predetermined position relative to a fastening point of an item present in the real space. The display device determines whether the work position is within the workable range, and if the work position is outside the workable range, outputs a first alert. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic view illustrating a display device according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. [Figure 3] FIG. 3 is a schematic diagram for explaining a display example on the display device. [Figure 4] FIG. 4 is a schematic diagram for explaining a display example on the display device. [Figure 5] 5(a) to 5(c) are schematic diagrams illustrating the working positions. [Figure 6] Fig. 6(a) is a schematic diagram for explaining processing by the display device according to the embodiment, and Fig. 6(b) is a schematic diagram showing a display example by the display device according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 10] 10(a) and 10(b) are schematic diagrams showing examples of display by the display device according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example of a tool. [Figure 12] FIG. 12 is a schematic diagram showing the state of the work. [Figure 13] FIG. 13 is a schematic diagram for explaining the calculation method in the first embodiment. [Figure 14] 14(a) and 14(b) are schematic diagrams for explaining the calculation method in the first embodiment. [Figure 15] FIG. 15 is a schematic diagram for explaining the calculation method in the first embodiment. [Figure 16]FIG. 16 is a schematic diagram for explaining the calculation method in the first embodiment. [Figure 17] FIG. 17 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 18] FIG. 18 is a schematic diagram showing a specific example of a virtual object. [Figure 19] FIG. 19 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 21] FIG. 21 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 22] FIG. 22 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 23] FIG. 23 is a schematic diagram showing a display example on the display device according to the embodiment. [Figure 24] FIG. 24 is a flowchart illustrating a processing method according to an embodiment. [Figure 25] FIG. 25 is a flowchart showing a checking method according to an embodiment. [Figure 26] FIG. 26 is a schematic diagram showing the configuration of an acquisition system according to an embodiment. [Figure 27] FIG. 27 is a schematic diagram for explaining the process in the workable range registration mode. [Figure 28] FIG. 28 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those already described are designated by the same reference numerals, and detailed description will be omitted as appropriate.

[0008] FIG. 1 is a schematic view illustrating a display device according to an embodiment. 1 , a display device 100 according to an embodiment includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a light source 133, an eye tracking camera 134, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0009] In the illustrated example, the display device 100 is a twin-eye head-mounted display. Two lenses, 111 and 112, are fitted into a frame 101. Projection devices 121 and 122 project information onto the lenses 111 and 112, respectively.

[0010] Projection device 121 and projection device 122 display the results of body recognition of the worker (wearer), virtual objects, etc. on lens 111 and lens 112. Only one of projection device 121 and projection device 122 may be provided, and information may be displayed on only one of lens 111 and lens 112.

[0011] Lenses 111 and 112 are optically transparent. The worker can see the real situation through lenses 111 and 112. The worker can also see information projected onto lenses 111 and 112 by projection devices 121 and 122. Information (virtual space) is displayed superimposed on real space through projection by projection devices 121 and 122.

[0012] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 emits infrared light and obtains a depth image based on the reflected infrared light. The light source 133 emits light (e.g., infrared light) toward the wearer's eyes. The eye tracking camera 134 detects the light reflected by the wearer's eyes. The sensor 140 is a six-axis detection sensor that can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts audio input.

[0013] The processing device 150 controls each element of the display device 100. For example, the processing device 150 controls the projection device 121 and the projection device 122 to display information on the lenses 111 and 112. Hereinafter, the processing device 150 causing the projection devices 121 and 122 to display information on the lenses 111 and 112 will also be simply referred to as "the processing device displays information." In addition, the processing device 150 detects movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection devices 121 and 122 in accordance with the movement of the field of view.

[0014] Additionally, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc. For example, the processing device 150 recognizes a preset object from the image obtained by the image camera 131. The processing device 150 recognizes the surface shape of the object from the image obtained by the depth camera 132. The processing device 150 calculates the viewpoint and line of sight of the worker's eyes from the detection results obtained by the eye tracking camera 134.

[0015] The battery 160 supplies the power required for operation to each element of the display device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the display device 100 and may communicate with the processing device 150.

[0016] The display device is not limited to the illustrated example, and may be a single-lens head-mounted display, or may be a glasses-type display device as illustrated, or a helmet-type display device.

[0017] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. For example, work is performed on an article 200 shown in FIG. 2. The article 200 is a cylindrical hollow member and has fastening points 201 to 204. In the work, a fastener such as a screw is fastened to the article using a tool. Alternatively, a screw fastened to the article is loosened using a tool. The article is a part, unit, or semi-finished product for manufacturing a product. The tool is a wrench, a screwdriver, or the like. Here, an example in which an embodiment of the present invention is applied to a fastening work of tightening a screw will be mainly described.

[0018] The worker uses an extension bar and a wrench to turn screws into each of the fastening points 201 to 204. A marker 210 is provided near the work object. In the illustrated example, the marker 210 is an AR marker. As will be described later, the marker 210 is provided to set the base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, the base point may be indicated by a hand gesture. The processing device 150 sets the three-dimensional coordinate system based on multiple points indicated by the hand gesture. For example, the three-dimensional coordinate system is represented by an X-axis direction (first axis), a Y-axis direction (second axis), and a Z-axis direction that are orthogonal to each other.

[0019] 3 and 4 are schematic diagrams for explaining examples of display on the display device. When the fastening operation starts, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position of the marker 210.

[0020] As long as a three-dimensional coordinate system can be set, any object can be used for the setting. Here, an example will be described in which a three-dimensional coordinate system is set using a marker 210. At the start of the work, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a base point in the virtual space based on the position and orientation of the marker 210. The three-dimensional coordinate system is determined based on the base point. By setting the base point based on an object that exists in real space, it becomes possible to display a virtual object that corresponds to the object in real space.

[0021] The image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand, and the worker's right hand. The processing device 150 recognizes the left hand and the right hand from the captured images. After recognizing the left hand 261 and the right hand 262, the processing device 150 measures the position of each hand. Specifically, the hand includes multiple joints, such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may also be used as the position of the hand. Alternatively, the position of the center of gravity of the entire hand may also be used as the position of the hand. The processing device 150 performs hand tracking, which repeatedly measures the position of the hand.

[0022] Processing device 150 causes projection device 121 and projection device 122 to display the recognition results on lenses 111 and 112. Hereinafter, the processing device causing information to be displayed on the lenses using the projection device will also be simply referred to as "the processing device displays information."

[0023] 3, the processing device 150 displays the recognition results of the left hand 261 and the right hand 262 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 261a and a plurality of virtual objects 262a are displayed as the recognition results of the left hand 261 and the right hand 262. The plurality of virtual objects 261a represent the plurality of joints of the left hand 261, respectively. The plurality of virtual objects 262a represent the plurality of joints of the right hand 262, respectively. Instead of the joints, virtual objects representing the surface shapes of the left hand 261 and the right hand 262, respectively, may be displayed.

[0024] The processing device 150 further calculates the position and orientation of the display device 100. As an example, the processing device 150 calculates the position and orientation of the display device 100 using a spatial mapping function. More specifically, the depth camera 132 measures the distance between the display device 100 and surrounding objects. Surface information of the surrounding objects is obtained from the measurement results (depth image) by the depth camera 132. The surface information includes the position and orientation of the object's surface. For example, the surface of each object is represented by multiple meshes, and the position and orientation are calculated for each mesh. The processing device 150 calculates the relative position and orientation of the display device 100 with respect to the surfaces of the surrounding objects from the surface information. When the markers 210 are recognized, the position of each surface is also represented in a three-dimensional coordinate system with the markers 210 as the base point. The position and orientation of the display device 100 in the three-dimensional coordinate system are calculated from the positional relationship between the surface of the object and the display device 100. Here, the direction of the display device 100 refers to the direction in front of the display device 100. For example, when a worker wears the display device 100, the direction of the display device 100 is parallel to the front direction of the worker's face.

[0025] Spatial mapping is repeatedly performed at predetermined intervals. Each time spatial mapping is performed, surface information of surrounding objects is obtained. The processing device 150 calculates changes in the position and orientation of the surface between the latest spatial mapping result and the previous spatial mapping result. In a situation where surrounding objects are not moving, changes in the position and orientation of the surface correspond to changes in the position and orientation of the display device 100, respectively. The processing device 150 calculates the amount of change in the position and orientation of the display device 100 from the changes in the position and orientation of the surface. The detection results of the sensor 140 may also be used to calculate the amount of change in the position and orientation of the display device 100. The processing device 150 updates the position and orientation of the display device 100 based on the obtained amount of change. Note that, instead of spatial mapping, an existing positioning method may be used to acquire the position and orientation of the display device 100.

[0026] When the position and orientation of the display device 100 are acquired by any of the methods, the processing device 150 sets a workable range using the acquired position and orientation. The workable range is a range in which a person wearing the display device 100 can work safely. When the workable range is set, the processing device 150 may display a virtual object 301 (first virtual object) indicating the workable range, as shown in FIG. 4. The worker can visually recognize the range in which they can work safely from the display of the virtual object 301.

[0027] 5(a) to 5(c) are schematic diagrams illustrating the working positions. The processing device 150 acquires the work position. The work position is set at a predetermined position relative to the fastening point of the item 200. The positional relationship of the work position relative to the fastening point is set appropriately depending on the work to be performed, the ease of setting the work position, etc. For example, the positions of the fastening points 201 to 204 are registered in advance in a database. The positions of the fastening points 201 to 204 are expressed using a three-dimensional coordinate system based on the marker 210. In this case, as shown in FIG. 5(a), the processing device 150 can use the position P1 of the fastening point 201 as the work position.

[0028] When data on the fastener to be used is registered in advance in addition to the positions of the fastening points 201 to 204, the processing device 150 may use position P2 as the work position, as shown in Fig. 5(b). Position P2 is located at the head of the screw 215 and is separated from the fastening point 201 by the length of the screw 215 in the direction of the screw hole.

[0029] In addition to the positions of the fastening points 201 to 204, data on the tools to be used may be registered in advance. For example, as shown in FIG. 5(c), when a wrench 251 and an extension bar 252 are used for work, the processing device 150 may use position P3 as the work position, as shown in FIG. 5(c). Position P3 is separated from the fastening point 201 by the length of the tool (extension bar 252) in the direction of the screw hole. Position P3 is the position where the hands should be placed during work. Note that when the extension bar 252 is not used and the screw is turned with the wrench 251, position P3 may be the same as position P2.

[0030] Fig. 6(a) is a schematic diagram for explaining processing by the display device according to the embodiment, and Fig. 6(b) is a schematic diagram showing a display example by the display device according to the embodiment. Once the work position is set, the processing device 150 determines whether the work position is within the workable range. In the example shown in Fig. 6(a), all of positions P1 to P3 are located outside the workable range indicated by the virtual object 301. Therefore, it is determined that the work position is not within the workable range.

[0031] As shown in FIG. 6(b), the processing device 150 outputs an alert 351 (first alert) indicating the danger of the work. The alert 351 includes messages 351a and 351b. The message 351a indicates the danger if the work is performed at the worker's current position. The message 351b indicates instructions to the worker. In response to the output of the alert 351, the worker approaches the fastening point 201.

[0032] Instead of a message such as alert 351, sound, light, vibration, or the like may be output as an alert. For example, processing device 150 may output a sound indicating the danger and instructions to the worker by voice. In addition to the message, sound, light, vibration, or the like may also be output.

[0033] The workable range for the position and orientation of the display device 100 is registered in advance for each worker. The processing device 150 refers to the workable range of the worker (the person wearing the display device 100). The processing device 150 sets the workable range based on the acquired position and orientation of the display device 100.

[0034] Alternatively, the workable range may be set using the worker's physique data. For example, the distance to the farthest hand within a safe work range relative to the position and orientation of the display device 100 is registered in advance as physique data. This distance depends on the wearer's arm length, neck length, shoulder width, etc. The processing device 150 refers to the distance registered in the wearer's physique data. The processing device 150 sets the range up to this distance as the workable range, based on the position and orientation of the display device 100.

[0035] More detailed physique data, such as the arm length from shoulder to hand, shoulder width, and neck length, may be registered. In this case, the processing device 150 can calculate the shoulder position using the position and orientation of the display device 100, the neck length, and the shoulder width. The processing device 150 uses the range set based on the arm length, with the shoulder position as a reference, as the workable range. Note that it is difficult for a worker to move a tool when their arm is stretched out straight. Therefore, it is preferable to use a value obtained by multiplying the arm length by a predetermined ratio as the distance from the display device 100 to the outer edge of the workable range.

[0036] 7 and 8 are schematic diagrams showing examples of display by the display device according to the embodiment. 3, the shape of the workable area (virtual object 301) may simply be a circle or a sphere. The center of the circle or sphere is the display device 100, and the radius of the circle or sphere is the pre-registered distance between the display device 100 and the hand.

[0037] Alternatively, as shown in Fig. 7, the shape of the workable area may be elliptical. This is because it is easiest for a worker to work in the direction (front) in which the display device 100 is facing. In this case, the dimension of the workable area in the direction of the display device 100 (front direction of the worker) is longer than the dimension of the workable area in an orthogonal direction perpendicular to the direction of the display device 100. For example, the processing device 150 uses a pre-registered distance between the display device 100 and the hand as the dimension of the workable area in the direction of the display device 100. The processing device 150 multiplies the distance by a predetermined ratio and uses the value obtained as the dimension of the workable area in the orthogonal direction.

[0038] Alternatively, the shape of the workable area may be a two-dimensional or three-dimensional torus. In other words, areas that are too close to the worker may not be included in the workable area. This is because areas that are too close to the worker have difficulty moving their arms and working. In this case, the workable area has two types of outer edges. For example, as shown in FIG. 8 , virtual objects 301a and 301b are displayed. Virtual object 301a indicates the inner outer edge of the workable area. Virtual object 301b indicates the inner outer edge of the workable area. The distance between the display device 100 and virtual object 301b is longer than the distance between the display device 100 and virtual object 301a. Virtual object 301a, located close to the worker, indicates the limit of the safe workable area. Virtual object 301b, located farther from the worker, indicates the limit of the safe workable area.

[0039] The advantages of the embodiment will be described. It is important that work is performed safely. For example, if the worker's position is far from the fastening point, the worker must stretch their arms to the limit. Also, they must bend forward to get their hands closer to the fastening point. If work is performed in such an inappropriate posture, there is a risk of falling and getting injured or straining their joints.

[0040] According to the embodiment, when the position and orientation of the display device 100 are acquired, the position and orientation are used to set the workable range. Also, a work position is set at a predetermined position relative to the fastening location. The processing device 150 determines whether the work position is within the workable range, and outputs an alert if the work position is outside the workable range.

[0041] The workable range corresponds to the range within which the wearer can work safely. If the work position is outside the workable range, an alert is output, allowing the worker to understand that the current position is inappropriate for the work. For example, the worker approaches the fastening point in response to the output of the alert. As a result, the work position will be within the workable range. The worker can perform the work safely in a more appropriate posture.

[0042] Furthermore, by displaying a virtual object indicating the workable range, the worker can grasp the workable range. When an alert is output, the worker can easily grasp how far they should move. This improves the usability of the display device 100.

[0043] According to an embodiment of the present invention, a display device is provided that allows work to be performed more safely and has better convenience.

[0044] 9, 10(a), and 10(b) are schematic diagrams showing examples of display by the display device according to the embodiment. A virtual object other than the virtual object 301 may be displayed. For example, as shown in Fig. 9, the processing device 150 displays a virtual object 311 at the fastening location 201. The virtual object 311 is displayed at the fastening location where work is to be performed. By displaying the virtual object 311, the worker can easily grasp the fastening location where work is to be performed.

[0045] The display position of the virtual object 311 may be registered in advance, or may be calculated using the position of the fastening location 201. For example, the virtual object 311 is displayed superimposed on the fastening location 201. In that case, the position of the fastening location 201 is used as the display position of the virtual object 311. The virtual object 311 may be slightly separated from the fastening location 201 in the direction of the screw hole. In that case, a display position separated from the fastening location 201 is registered in advance. Alternatively, if data of the fastener is registered in advance, the display position of the virtual object 311 may be calculated using the position of the fastening location 201 and the length of the fastener. If the display position of the virtual object 311 is registered in advance, the display position of the virtual object 311 may be used as a work position.

[0046] 10(a), the processing device 150 may display a virtual object 321a (second virtual object) and a virtual object 321b. The virtual object 321a is displayed at a position away from the fastening point 201 in the direction of the screw hole. The virtual object 321b is displayed between the fastening point 201 and the virtual object 321a. The virtual object 321b indicates which fastening point the virtual object 321a corresponds to.

[0047] Virtual object 321a indicates the position where the hand should be placed when turning a screw into fastening point 201. Virtual object 321b indicates the position where the extension bar should be placed when turning a screw into fastening point 201. For example, the distance between fastening point 201 and virtual object 321a corresponds to the length of the extension bar.

[0048] In the illustrated example, virtual object 321a is spherical, and virtual object 321b is rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each virtual object is not limited to this example. For example, virtual object 321a may be a cube, and virtual object 321b may be linear.

[0049] The worker positions the extension bar 252 so that the extension bar 252 is close to or in contact with the virtual object 321b. The worker also grips the head of the wrench 251 so that his or her hand comes into contact with the virtual object 321a. By displaying the virtual objects 321a and 321b, the worker can easily understand the positions of the tool and hand when turning the screw into the fastening point 201. This can improve work efficiency.

[0050] The display positions of the virtual objects 321a and 321b may be registered in advance, or may be calculated from the position of the fastening point 201 and tool data. When the display position of the virtual object 321a is registered in advance, the display position of the virtual object 321a may be used as the work position.

[0051] After the virtual object 321a is displayed, the processing device 150 may determine whether a predetermined object has come into contact with the virtual object 321a. For example, the processing device 150 determines whether a hand has come into contact with the virtual object 321a. Specifically, the processing device 150 calculates the distance between the position of the hand and the position of the virtual object 321a. If the distance is less than a preset threshold, the processing device 150 determines that the hand has come into contact with the virtual object 321a. As an example, in FIG. 10(a), the diameter of the virtual object 321a (sphere) corresponds to the threshold. The sphere indicates the range within which it is determined that the hand has come into contact with the virtual object 321a.

[0052] FIG. 11 is a schematic diagram showing an example of a tool. The processing device 150 may determine whether the tool has come into contact with the virtual object 321a. For example, as shown in FIG. 11 , a wrench 251 has multiple markers 251a attached thereto. The processing device 150 recognizes the multiple markers 251a from an image captured by the image camera 131. The processing device 150 measures the position of each marker 251a. The positional relationship between the multiple markers 251a and the head 251b of the wrench 251 is registered in advance. The processing device 150 calculates the position of the head 251b based on the positions of at least three recognized markers 251a and the previously registered positional relationship. The processing device 150 calculates the distance between the position of the head 251b and the position of the virtual object 321a. If the distance is less than a preset threshold, the processing device 150 determines that the wrench 251 has come into contact with the virtual object 321a.

[0053] When a predetermined object comes into contact with the virtual object 321a, it can be estimated that a screw is being turned into the fastening point 201 corresponding to the virtual object 321a.

[0054] In addition to contact between an object and a virtual object, the fastening location being worked on may also be estimated using hand movement. While a screw is being turned with a tool, the hand turning the tool moves in an arc. At this time, the position of the center of rotation remains almost constant. For example, while a screw is being turned with a wrench, the position of the wrench head remains almost constant. When the change in the position of the center of rotation becomes small, it can be estimated that the screw is being turned.

[0055] FIG. 12 is a schematic diagram showing the state of the work. For example, as shown in FIG. 12 , a worker uses a wrench 251 to tighten a screw at a fastening point. Here, an example will be described in which an extension bar 252 is not used. The worker places a screw 215 in a screw hole at a fastening point (not shown). The worker holds the grip of the wrench 251 with his right hand and inserts the tip (head) of the wrench 251, to which a socket is attached, into the screw 215. The worker rotates the wrench 251 to turn the screw 215.

[0056] While the worker turns the wrench 251, the processing device 150 repeatedly measures the position of the hand. At this time, the hand is positioned on a circumference with a part of the wrench 251 as its center. The hand is moved in an arc. The processing device 150 uses this to estimate the position of the center of rotation of the tool. Based on the position of the center of rotation, the processing device 150 estimates the fastening point where the screw is being turned, the execution of work on the fastening point, etc. For example, the following first or second calculation method is used to calculate the center position.

[0057] 13, 14(a), 14(b), 15, and 16 are schematic diagrams for explaining the calculation method in the first embodiment. In the first calculation method, the processing device 150 extracts three mutually different positions from the multiple measured positions. The processing device 150 calculates the circumcenter O of the three positions. Here, as shown in FIG. 15 , the three positions are defined as P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3), respectively. The position of the circumcenter O is defined as P0 (x0, y0, z0). Furthermore, for a triangle obtained by connecting positions P1 to P3, the length of the opposite side of position P1 is defined as L1. The length of the opposite side of position P2 is defined as L2. The length of the opposite side of position P3 is defined as L3. The angle at position P1 is defined as α. The angle at position P2 is defined as β. The angle at position P3 is defined as γ. In this case, the position of the circumcenter O is expressed by the following mathematical formula (1). Note that in mathematical formula (1), symbols with arrows indicate position vectors. Equation (1) can be rewritten as equation (2). Equation (2) can be decomposed into equations (3) to (5).

number

number

number

number

number

[0058] From the formulas (3) to (5), x0, y0, and z0 are calculated. The processing device 150 calculates the position P0 (x0, y0, z0) of the circumcenter O as the center position of the rotation of the wrench 251.

[0059] The center position of the rotation of the wrench 251 can be considered to be the position at which the screw 215 is turned by the wrench 251. Then, it can be estimated that the screw is being tightened at the fastening point closest to the center position. For example, the processing device 150 extracts combinations of three positions from multiple hand positions measured over a predetermined time interval and calculates the center position. The processing device 150 repeats the extraction of position combinations and the calculation of the center position. The processing device 150 calculates the distance between each center position calculated over that time interval and the fastening point, and if any of the distances is less than a threshold, it estimates that the screw is being tightened at that fastening point.

[0060] When a digital tool capable of detecting torque values ​​is used, the detection results by the tool may be used to estimate the operation. For example, if the distances between each center position and the fastening point within a predetermined time span are all less than a threshold value and a torque value is detected by the tool, the processing device 150 estimates that a screw is being turned into the fastening point.

[0061] In the first calculation method described above, the length of a tool interposed between the wrench 251 and the screw 215 may be used in the calculation to more accurately estimate the position of the screw. In the example shown in FIG. 12 , a socket 253 is fitted to the wrench 251. That is, the center position of rotation of the wrench 251 and the position of the screw 215 are separated by the length of the socket 253. If the length of the socket 253 is registered in advance, the processing device 150 can more accurately estimate the position of the screw 215 using the center position and the length of the socket 253.

[0062] When estimating the position of the screw 215 using the length of the socket 253, it is necessary to determine on which side the screw 215 is located with respect to the plane on which the wrench 251 is rotating. In the example shown in FIG. 14(a), the wrench 251 is rotating in a rotation direction RD1. The screw 215 and the socket 253 are located on the lower side. In the example shown in FIG. 14(b), the wrench 251 is rotating in a rotation direction RD2. The rotation direction RD2 is opposite to the rotation direction RD1. The screw 215 and the socket 253 are located on the upper side with respect to the plane parallel to the rotation direction RD2.

[0063] To determine the side on which the screw 215 is located, the processing device 150 uses the center position, two positions of the hand, time-series information on the two positions, and screw loosening information. For example, as shown in FIG. 27, the two positions are P1 (x1, y1, z1) and P2 (x2, y2, z2), respectively. The center position is P0 (x0, y0, z0). The time when the hand was at position P1 and the time when the hand was at position P2 are known. That is, the processing device 150 holds time-series information on positions P1 and P2. In this example, the time when the hand was at position P1 was before the time when the hand was at position P2.

[0064] The loosening / reducing information indicates whether the screw is being tightened or loosened. If the wrench 251 is a digital tool, the wrench 251 determines whether the screw is being tightened or loosened from the detected torque value and generates the loosening / reducing information. The processing device 150 may determine whether the screw is being tightened or loosened from the time-series data of the torque value received from the wrench 251 and generate the loosening / reducing information.

[0065] The plane passing through the positions P0 to P2 is expressed by the following formula (6): In formula (6), k, l, m, and n are constants. [Number 6] kx+ly+mz+n=0

[0066] By substituting the coordinates of P0 to P2 into equation (6), the following equations (7) to (9) are obtained: From equations (7) to (9), the constants k, l, m, and n are calculated. [Number 7] kx0+ly0+mz0+n=0 [Number 8] kx1+ly1+mz1+n=0 [Number 9] kx2+ly2+mz2+n=0

[0067] Here, the processing device 150 calculates a vector from the center position P0 to a position P1 at an earlier time. The processing device 150 also calculates a vector from the center position P0 to a position P2 at a later time. The screw 215 is located at a position P0 on the normal vector P0P1×P0P2, which is a distance L0 from the center position P0. Q When the screw is tightened and the time at position P1 is earlier than the time at position P2, the processing unit 150 calculates a normal vector P0P1×P0P2 between vector P0P1 and vector P0P2.

[0068] From position P0, position P where the wrench and socket act on the screw Q The length to is expressed by the following formula (10): In the formulas below, an arrow attached to a symbol indicates that the value indicated by the symbol is a vector.

number

[0069] On the other hand, from position P0 to position P Q The vector up to can also be expressed by the following equation (11): In equation (11), t is a constant.

number

[0070] By substituting Equation (11) into Equation (10), the following Equation (12) is obtained. The length L0 in Equation (12) is registered in advance. By solving Equation (12), t is calculated.

number

[0071] Once t is calculated, the position P is calculated using the position P0, constants k, l, m, n, and t. Q is calculated, i.e., the position of the screw is obtained.

[0072] For example, the processing device 150 extracts a combination of three positions from the multiple positions of the screw 215 calculated over a predetermined time interval, and calculates the center position. The processing device 150 repeats the extraction of the combinations of positions and the calculation of the center position. If the distance between each position of the screw 215 and the fastening point over that time interval is less than a threshold, the processing device 150 estimates that the screw is being turned with respect to that fastening point.

[0073] As shown in FIG. 5(c), there is a case where the screw is fastened via an extension bar 252. In this case, the position of the screw 215 can be estimated using the length of the extension bar 252, as in the method described above. That is, the screw 215 is located on the normal vector P0P1×P0P2 at a distance from the center position P0 by the sum of the length of the extension bar 252 and the length of the socket 253. The position P0 of the screw 215 can be calculated using the center position P0, the length of the extension bar 252, and the length of the socket 253. Q By taking into consideration the length of another tool interposed between the screw 215 and the wrench 251, the position of the screw 215 can be estimated with higher accuracy.

[0074] In the second calculation method, the center position of rotation for each fastening point is registered in advance. For example, as shown in FIG. 16, center positions c1 and c2 are registered in advance for fastening points 201 and 202. Three positions p1 to p3 of the hand are calculated from an image of the hand turning the screw. The processing device 150 calculates distances d1 to d3 between the center position c1 and positions p1 to p3, respectively. Similarly, the processing device 150 calculates distances d1 to d3 between the center position c1 and positions p1 to p3, respectively. The processing device 150 calculates the variance of each distance between the center position c1 and positions p1 to p3, and calculates the variance of each distance between the center position c2 and positions p1 to p3. If any of the variances is less than a threshold, the processing device 150 estimates that the tool is rotating at that center position. The processing device 150 estimates that the screw is being turned with respect to the fastening point associated with the estimated center position.

[0075] The variation may be the sum of the differences between the average value of multiple distances and each distance, the variance of multiple distances, or the standard deviation of multiple distances. In the example shown in Fig. 16, hand positions p1 to p3 are approximately equidistant from the center position c1. Therefore, it is estimated that the screw is being turned to the fastening point 201 associated with the center position c1.

[0076] When a digital tool is used, the detection results of the tool may be used to estimate the operation. For example, if the center position is estimated and the torque value is detected by the tool, the processing device 150 estimates that a screw is being turned into the fastening point associated with the center position.

[0077] The processing device 150 may repeat the first or second calculation method regardless of whether an operation is being performed. Specifically, the processing device 150 executes the first or second calculation method using multiple hand positions obtained over a predetermined time span. If the execution of an operation is not estimated from the multiple hand positions over that time span, the processing device 150 slides the time span and executes the first or second calculation method again. As an example, the time span is set to 6 seconds, and the slide amount is set to 16 milliseconds. The time span and slide amount are set appropriately depending on the performance of the processing device 150.

[0078] When no work is being performed, the tool is not actually rotating and there is no center of rotation. However, it is possible to calculate the apparent center position from multiple positions of the hand. While no work is being performed, the calculated center position is far from the position of the fastening point. Therefore, it is not estimated that work is being performed. When work is being performed, the calculated center position approaches the position of the fastening point. The processing device 150 estimates the start of work as the time when it is first estimated that work is being performed.

[0079] When the fastening location being worked on is estimated from the hand movement or contact between the virtual object and a predetermined object, the processing device 150 may associate a work record with the data of the fastening location. The work record indicates that a screw has been turned into the fastening location. This allows the work record to be created automatically.

[0080] When a digital tool such as a digital torque wrench or digital torque driver is used, the processing device 150 receives the detected torque value from the tool. The torque value required for fastening may be set in advance, and the digital tool may determine whether the required torque value has been detected. The digital tool transmits the determination result to the processing device 150. The digital tool also transmits the rotation angle, the time when the torque value was detected, and other information to the processing device 150. For example, the processing device 150 links the received maximum torque value or the determination result to data on the work location. This allows a more detailed work record to be automatically created.

[0081] The processing device 150 may determine whether work on the estimated fastening location has been completed based on the received torque value. If the received torque value is equal to or greater than a preset torque value, the processing device 150 determines that work on the fastening location has been completed. When the processing device 150 receives a determination result indicating whether the required torque value has been detected from the tool, the processing device 150 may determine whether work on the fastening location has been completed based on the determination result.

[0082] If, after it is estimated that work will be performed on the fastening point, the state in which work on the fastening point cannot be estimated or a torque value is not received continues for a predetermined period of time or more, the processing device 150 may determine that work on the fastening point has been completed.

[0083] A screw may be tightened multiple times at one fastening point. For example, after a screw is tightened at each of the fastening points 201 to 204, the screw is tightened again at each fastening point. In this case, the processing device 150 may count the number of times the screw is tightened. After it is estimated that the work is being performed, the processing device 150 counts the number of times the screw is tightened when it is determined that the work is completed.

[0084] 17 and 19 to 23 are schematic diagrams showing examples of display by a display device according to an embodiment. Fig. 18 is a schematic diagram showing a specific example of a virtual object. As shown in FIG. 17 , the processing device 150 may display a virtual object 331 (third virtual object). The virtual object 331 includes information about the work to be done at the fastening point 201. The virtual object 331 is displayed near the fastening point 201. For example, the distance between the fastening point 201 and the virtual object 331 is shorter than the distance between the other fastening points 202 to 204 and the virtual object 331. The information on the virtual object 331 is represented using characters (logographic characters, phonetic characters, or ideographic characters). The worker can understand the information necessary for the work from the virtual object 331.

[0085] 18, for example, virtual object 331 includes task information such as identification information 331a, specified torque value 331b, detection value 331c, meter 331d, ratio 331e, and number of times 331f. Identification information 331a is unique identification information assigned to fastening point 201 and is expressed as a character string. Specified torque value 331b is a torque value required to fasten a screw to fastening point 201 and is specified in advance.

[0086] A tool capable of detecting a torque value may be used in the work. In that case, detected value 331c indicates the torque value detected by the tool. Meter 331d indicates the specified torque value and the detected torque value. Ratio 331e indicates the ratio of the detected value to the specified torque value. Some work requires fastening a screw multiple times into one fastening point. In that case, count 331f indicates the number of times the screw has been fastened into fastening point 201. The worker performs the work while checking the content displayed in virtual object 331.

[0087] When the order of work on multiple fastening locations is determined, the display of virtual object 331 indicates the fastening locations to be worked on. From virtual object 331, the worker can check work information and also understand the fastening locations to be worked on.

[0088] 19, the processing device 150 may display virtual objects 331 to 334 at the fastening points 201 to 204, respectively. The virtual objects 331 to 334 include information about the work to be done at the fastening points 201 to 204, respectively. By displaying the virtual objects 331 to 334, the worker can check the information about the work to be done at each fastening point at once.

[0089] The display modes of the virtual objects 331 to 334 may indicate the fastening locations to be worked on, the number of times the screws have been tightened, etc. In the example shown in FIG. 20 , work is next performed on the fastening location 202. The virtual object 332 displayed at the fastening location 202 is displayed darker than the other virtual objects 331, 333, and 334. The colors of the virtual objects 331 to 334 change depending on the number of times the screws have been tightened. In the example shown, a screw has been tightened once at the fastening locations 202 to 204, and a screw has been tightened twice at the fastening location 201. Therefore, the color of the fastening location 201 is different from the colors of the fastening locations 202 to 204.

[0090] The fastening locations to be worked on and the number of times the screws are to be tightened may be indicated by size, changes in display (animation), etc., in addition to the density and color of the display. Virtual objects indicating the fastening locations to be worked on, such as virtual objects 321a and 321b, may also be displayed.

[0091] After all work is completed, the processing device 150 may determine whether the work on the fastening points 201-204 was performed properly. The processing device 150 refers to the records created during the work on the fastening points 201-204. The processing device 150 determines whether the maximum torque value received during the work on each fastening point is equal to or greater than a predetermined torque value. The processing device 150 also determines whether the number of times the screw has been tightened is equal to or greater than a predetermined number for each fastening point. If the received torque value is less than the predetermined torque value or if the number of times the screw has been tightened is less than the predetermined number, the processing device 150 outputs an alert.

[0092] For example, the fastening points 201 to 204 are each configured to be screwed twice. The fastening points 201 to 203 are each screwed twice, and the fastening point 204 is screwed only once. In this case, as shown in FIG. 21 , the processing device 150 changes the display mode of the virtual object 334 corresponding to the fastening point 204 from the display mode of the virtual objects 331 to 333. The border of the virtual object 334 is displayed thicker than the borders of the virtual objects 331 to 333, emphasizing the virtual object 334. The change in the display mode of the virtual object 334 functions as an alert (an example of a second alert). The processing device 150 may also display an alert 352 (another example of a second alert) including a message. A similar alert is output when the torque value for any of the fastening points is less than a predetermined torque value.

[0093] Instead of or in addition to a display, sound, light, vibration, or the like may be output as an alert. The output of the alert allows the worker to understand that the work on any of the fastening locations is inappropriate. For example, the worker checks which fastening location the work on is inappropriate from the information included in the virtual objects 331 to 334. The worker redoes the work on any of the fastening locations. This can prompt the worker to perform the appropriate work.

[0094] When a work order for multiple fastening locations is defined, the processing device 150 can also determine whether the fastening location being worked on is appropriate. For example, the processing device 150 determines the next fastening location to be worked on based on a record of work performed up to that point. The processing device 150 determines whether the fastening location estimated to be worked on matches the next fastening location to be worked on. If it is estimated that work is being performed on a fastening location different from the fastening location to be worked on, the processing device 150 may display an alert 353 as shown in FIG. 22. Instead of or in addition to a message such as the alert 353, sound, light, vibration, or the like may be output as an alert.

[0095] 23, virtual objects 341 and 342 may be displayed. Virtual object 341 is displayed to start processing by the processing device 150. Virtual object 342 is displayed to end processing by the processing device 150. For example, a three-dimensional coordinate system is set by the processing device 150. Thereafter, when the worker is ready to work, he or she touches virtual object 341 with his or her finger. When the processing device 150 determines that the worker's hand has touched virtual object 341, it starts processing such as setting a workable range, setting a work position, and comparing the work position with the workable range.

[0096] A voice command or a hand gesture may be used instead of touching the virtual object 341. When the worker has completed preparations for the work, he or she speaks a voice command or makes a hand gesture with his or her hand. The processing device 150 detects the voice command from the voice acquired by the microphone 141. Alternatively, the processing device 150 detects the hand gesture from the result of hand tracking.

[0097] For example, the processing device 150 automatically repeats spatial mapping. Furthermore, when the processing device 150 recognizes the marker 210 from the image, it automatically sets a three-dimensional coordinate system. Therefore, once the work to be performed is determined and the three-dimensional coordinate system is set, processing such as setting the workable range, setting the work position, and comparing the work position with the workable range can be performed. For example, if these processing operations are performed while the worker is preparing for work, the work position may be determined to be outside the workable range. During work preparation, alerts may be continuously output, which may disrupt the work preparation.

[0098] To address this issue, an instruction indicating that work preparations are complete is input to the display device 100 by touching the virtual object 341, using a voice command, or using a hand gesture. The processing device 150 starts processing in response to the instruction. This makes it possible to avoid an alert being output unnecessarily when work preparations are not complete. This makes it less likely that an alert will hinder the worker's actions, improving the convenience of the display device 100.

[0099] When the worker completes the work, he or she touches the virtual object 342 with a finger. When the processing device 150 determines that the worker's hand has touched the virtual object 342, it ends the processing. For example, after the worker touches the virtual object 341, the fastening location where the work is being done is estimated. When the worker touches the virtual object 342, the processing device 150 determines that the work on the estimated fastening location has been completed. Instead of touching the virtual object 342, an instruction indicating the completion of the work may be input to the display device 100 by a voice command or a hand gesture.

[0100] FIG. 24 is a flowchart illustrating a processing method according to an embodiment. 24 is executed, task master data 170a, base point master data 170b, tool master data 170d, and fastening point master data 170e are prepared in advance. Each piece of master data is registered in the storage device 170.

[0101] First, a task to be performed is selected (step S1). The task master data 170a has registered therein a task ID, a task name, an item ID, and an item name. A task is identified by the task ID, the task name, the ID of the item to be worked on, or the name of the item. The processing device 150 accepts the selection of a task. For example, the task to be performed is selected by a worker. The task to be performed may be selected by a higher-level system, and the processing device 150 may accept the selection. The processing device 150 may determine the task to be performed based on data obtained from the image camera 131 or another sensor. The processing device 150 selects the task based on the result of this determination.

[0102] Next, the image camera 131 captures an image of the marker 210. The processing device 150 sets a base point of a three-dimensional coordinate system based on the position and orientation of the marker 210 (step S2). At this time, the processing device 150 refers to the base point master data 170b. The base point master data 170b registers a method for setting a base point for each task. The processing device 150 acquires the method for setting a base point for the selected task and sets the base point according to that setting method.

[0103] After setting the base point, the processing device 150 acquires (step S3) the position and orientation of the display device 100. The position and orientation are acquired using spatial mapping or other positioning systems.

[0104] The processing device 150 refers to the physique data registered in the physique master data 170c. For example, arm length, neck length, and shoulder width are registered as the physique data. Or, more simply, the maximum distance between the display device 100 and the hands within a safe workable range may be registered as the physique data. Alternatively, the workable range relative to the position and orientation of the display device 100 may be registered in advance. In this case, the pre-registered workable range is referenced instead of the physique data. The processing device 150 sets the workable range using the position and orientation of the display device 100 and any of the data. The processing device 150 also displays a virtual object indicating the workable range (step S4).

[0105] The processing device 150 sets the work position (step S5). When the work position is set, the tool master data 170d and the fastening point master data 170e are referred to as appropriate. For example, the position of the fastening point is set as the work position. Alternatively, the work position may be calculated using the position of the fastening point and the tool data.

[0106] The tool master data 170d registers, for each task, the ID of the tool to be used, the tool model, tool length, socket model, socket length, etc. The tool model indicates the classification of the tool based on structure, shape, performance, etc. The tool length is the length from the center of rotation to the grip when the tool is used to tighten a screw. The socket model indicates the classification of the socket based on structure or shape. The socket length indicates the length of the socket in the direction connecting the tool and the screw when the screw is tightened. The processing device 150 obtains data on the tools to be used in the task selected in step S1 from the tool master data 170d. If an extension bar is used, the model, length, etc. of the extension bar are also registered in the tool master data 170d. The processing device 150 further obtains data related to the extension bar from the tool master data 170d.

[0107] The fastening point master data 170e registers, for each fastening point, the fastening point ID, the position of the fastening point, the required torque value, and the number of times the screw is tightened. The fastening position indicates the position where the fastening point is located, and the coordinates of the three-dimensional coordinate system set in step S2 are registered. The number of times the screw is tightened indicates the number of times the screw is tightened that is required for each fastening point. If a mark is applied to the screw after tightening, the color of the mark is also registered.

[0108] The processing device 150 determines whether the work position is within the workable range (step S6). If the work position is outside the workable range, the processing device 150 outputs an alert (step S7). After the alert is output, step S3 is executed again. For example, the worker moves in response to the alert, and the position and direction of the display device 100 after the movement are acquired. A new workable range is set, and the display of the virtual object indicating the workable range is updated. If the work position is found to be within the workable range as a result, the processing device 150 stops the alert (step S8). If it is determined that the work position is within the workable range without an alert being output, step S8 is omitted.

[0109] The processing device 150 determines whether the work has been completed (step S9). As described above, the completion of the work is determined based on the estimated results of the work execution, data received from the tool, etc. If the work has not been completed, step S3 is executed again.

[0110] If it is determined that the work has been completed, the processing device 150 creates a work record for the fastening location where the work is estimated to have been completed (step S10). The created record is saved in the history data 170f. For example, the torque value detected by the tool is linked to the work ID and the ID of the estimated fastening location. As shown in the figure, the processing device 150 may further link the type and ID of the tool used, the number of times the screw was tightened, and the mark recognition result to the ID of the fastening location. The mark is recognized by the processing device 150 from an image captured by the image camera 131. The processing device 150 extracts a cluster of pixels of the mark color from the image and counts the number of pixels in that cluster. If the number of pixels exceeds a preset threshold, it is determined that a mark has been added.

[0111] The processing device 150 determines whether work has been completed for all fastening locations (step S11). If work has not been completed for all locations, step S3 is executed again. Thereafter, in step S5, a work position corresponding to the next fastening location is set.

[0112] The processing device 150 may record the comparison result between the work position and the workable range in the history data 170f. In the example shown in FIG. 24, the determination result is linked to the fastening location data as "posture." For example, even if the work position is outside the workable range and an alert is output, if the execution of the work is estimated and a preset torque value is detected, the work may be determined to be completed. In this case, the processing device 150 links first data indicating that the work was performed when the work position was outside the workable range to the fastening location data. The assignment of the first data indicates that the work was performed with an inappropriate posture. Furthermore, the processing device 150 may link second data to the fastening location data when the work was performed when the work position was outside the workable range. The assignment of the second data indicates that the work was performed with an appropriate posture.

[0113] By adding the first data, when the work record is reviewed, it is easy to confirm whether the worker is performing the work in an appropriate posture. For example, if a worker is injured, the cause can be more easily investigated by checking whether the first data is present in the work record.

[0114] Alternatively, if the first data is frequently assigned, the workable range may be too narrow. In this case, if the workable range is too narrow, the worker's position may be unnecessarily restricted, which may reduce work efficiency. Therefore, if the first data is frequently assigned, correction of the workable range may also be considered.

[0115] FIG. 25 is a flowchart showing a checking method according to an embodiment. After it is determined in step S11 that all work has been completed, a confirmation method M2 shown in Fig. 25 may be executed. First, the processing device 150 reads each master data (step S21). Data such as the work master data 170a, the tool master data 170d, and the fastening point master data 170e are read. Next, the processing device 150 reads the work record stored in the history data 170f (step S22).

[0116] The processing device 150 compares the information of the read master data with the information of the work record, and checks whether the work has been performed appropriately for each fastening point (step S23). As described above, if the received torque value for any fastening point is less than a predetermined torque value, or if the number of screw tightenings is less than a predetermined number, the work for that fastening point is determined to be inappropriate.

[0117] The processing device 150 determines whether any work has been determined to be inappropriate (step S24). If any inappropriate work has been determined, the processing device 150 outputs an alert (step S25). The worker performs the work again in response to the alert. The processing device 150 then determines whether the work has been completed for the fastening locations for which the work has been determined to be inappropriate (step S26). If the torque value is insufficient for any fastening location, the processing device 150 estimates the work for that fastening location and determines that the work has been completed when a torque value equal to or greater than a predetermined torque value is received from the tool. If the number of screw tightening operations is insufficient for any fastening location, the processing device 150 estimates the work for that fastening location and determines that the work has been completed when the number of screw tightening operations reaches a predetermined number.

[0118] If it is determined in step S24 that no inappropriate work exists, or if it is determined in step S26 that the work has been completed, the processing device 150 ends the confirmation process.

[0119] FIG. 26 is a schematic diagram showing the configuration of an acquisition system according to an embodiment. 26 acquires physique data referenced by a processing device 150. The acquisition system 1 includes an imaging device 2 and a processing device 3.

[0120] The imaging device 2 captures an image of the worker. The captured image shows at least the upper body. The imaging device 2 includes a camera. The processing device 3 receives the image captured by the imaging device 2. The processing device 3 inputs the image into a posture estimation model. When an image is input, the posture estimation model estimates the posture of the person depicted in the image. The posture is represented by the joints of the human body and the skeleton that connects the joints. The joints include the head, neck, shoulders, elbows, wrists, fingers, hips, knees, and ankles.

[0121] The pose estimation model preferably includes a neural network. More preferably, the pose estimation model includes a convolutional neural network (CNN). Examples of the pose estimation model that can be used include OpenPose, DarkPose, CenterNet, and the like.

[0122] The processing device 3 acquires the estimation result output from the posture estimation model. The processing device 3 calculates physique data such as arm length, shoulder width, and neck length from the estimated posture. The processing device 3 registers the calculated physique data in the physique master data 170c. By using the acquisition system 1, the physique data of the wearer of the display device 100 can be easily acquired.

[0123] The display device 100 may also have the function of the acquisition system 1. For example, a worker wearing the display device 100 stretches his / her hand as far as possible within the range where the worker can work safely. The processing device 150 measures the position of the hand from the image at that time. The processing device 150 also calculates the position and direction of the display device 100 at that time. The processing device 150 calculates the distance between the position of the display device 100 and the position of the hand, and registers the distance in the physique master data 170c.

[0124] FIG. 27 is a schematic diagram for explaining the process in the workable range registration mode. The display device 100 may be capable of executing a registration mode for registering the workable range. The worker wears the display device 100 and executes the registration mode. As shown in FIG. 27 , the worker stretches his / her arms to the limit of the range where work can be performed safely. The processing device 150 measures the positions of the left hand 261 and the right hand 262 at that time. The processing device 150 measures the distance from the position of the display device 100 to each hand. The processing device 150 calculates the workable range using the distances and registers the workable range.

[0125] Alternatively, the worker may move his / her hands within a range that is considered safe for work. The processing device 150 repeatedly measures the hand position and calculates the workable range indicated by the hand movement. The processing device 150 registers the obtained workable range.

[0126] For example, the display device 100 is an AR device that displays augmented reality (AR) or an MR device that displays mixed reality (MR). When the display device 100 is realized as an MR device, it can detect contact between a virtual object and a real object. Therefore, when the execution of a task is estimated based on contact between a predetermined object and a virtual object, it is preferable that the display device 100 is an MR device.

[0127] FIG. 28 is a schematic diagram showing a hardware configuration. 28 is used as the processing device 3 or the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0128] The ROM 92 stores a program that controls the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to perform each of the above-described processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0129] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0130] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may also be used as the storage device 170.

[0131] An input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0132] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can output data to the output device via the output I / F 96.

[0133] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0134] The data processing by the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0135] The various data processing operations described above may be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, or DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0136] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0137] Embodiments of the invention include the following features. (Feature 1) A display device that displays a virtual object superimposed on a real space, obtaining the position and orientation of the display device; displaying a first virtual object indicating a workable range set using the position and the direction; A work position is set at a predetermined position relative to a fastening point of an article existing in the real space; a display device that determines whether the work position is within the workable range, and outputs a first alert if the work position is outside the workable range. (Feature 2) Inferring whether work has been performed on the fastening point; The display device according to feature 1, wherein when it is estimated that the work has been performed, a record of the work is linked to the data of the fastening location. (Feature 3) Identifying the fastening points to be worked on based on the positions of the plurality of fastening points registered in advance and the records for each of the plurality of fastening points; The display device according to feature 2, wherein the work position is set at the predetermined position relative to the identified fastening location. (Feature 4) Receives torque values ​​from the tool that turns the screw, The display device according to feature 2, wherein the torque value is linked to the data of the fastening point as the record. (Feature 5) After the work, it is determined whether the received torque value is equal to or greater than a predetermined torque value, and whether the number of times the screw has been tightened at the fastening point matches a predetermined number of times; The display device according to feature 4 outputs a second alert when the received torque value is less than the predetermined torque value or when the number of screw tightenings is less than the predetermined number of times. (Feature 6) The display device according to any one of features 2 to 5, wherein when the work is performed with the work position outside the workable range, first data is linked to the data of the fastening location. (Feature 7) displaying a second virtual object in correspondence with the fastening location; The display device according to any one of features 2 to 6, wherein the execution of work on the fastening location is estimated based on contact between a predetermined object and the second virtual object. (Feature 8) Measure the hand position from the image, estimating a center of rotation position of a tool that turns a screw from the plurality of positions of the hand; The display device according to any one of features 2 to 6, wherein the execution of work on the fastening location is estimated based on the estimated center position. (Feature 9) displaying a third virtual object including information about the work to be performed on the fastening location; A display device described in any one of features 1 to 8, wherein the information includes one or more selected from a specified torque value required to tighten the screw into the fastening point, a torque value detected by a tool used to tighten the screw, and the number of times the screw is tightened into the fastening point. (Feature 10) Repeating the acquisition of the position and the orientation of the display device and the setting of the workable range; 10. The display device according to any one of features 1 to 9, wherein after the first alert is output, if it is determined that the work position is within the workable range, the first alert is stopped. (Feature 11) 11. The display device according to any one of Features 1 to 10, wherein the length of the workable range of the display device in the direction is longer than the length of the workable range in an orthogonal direction perpendicular to the direction. (Feature 12) A registration mode for registering the workable range can be executed, A display device according to any one of features 1 to 11, wherein in the registration mode, multiple positions of the hand are measured, the position of the display device is measured, and the workable range is registered based on the distance from the position of the display device to the hand. (Feature 13) 12. The display device according to any one of Features 1 to 11, wherein the workable range is set using the position and the orientation of the display device and physique data of a wearer of the display device. (Feature 14) 14. An acquisition system for acquiring the physique data referred to by the display device according to feature 13, An acquisition system that acquires the physique data from a distance between a position of the display device and a position of the wearer's hand or from an image of the wearer's body. (Feature 15) A display device that displays virtual objects overlaid on real space. obtaining a position and orientation of the display device; displaying a first virtual object indicating a workable range set using the position and the direction; A work position is set at a predetermined position relative to a fastening location of an article existing in the real space; determining whether the work position is within the workable range, and outputting a first alert if the work position is outside the workable range; Processing method. (Feature 16) A program that causes the display device to execute the processing method according to feature 15. (Feature 17) A storage medium storing the program according to feature 16.

[0138] According to the embodiments described above, a display device, an acquisition system, a processing method, a program, and a storage medium are provided that enable work to be performed more safely.

[0139] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0140] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other. [Explanation of symbols]

[0141] 1: Acquisition system, 2: Imaging device, 3: Processing device, 100: Display device, 101: Frame, 111, 112: Lens, 121: Projection device, 122: Projection device, 131: Image camera, 132: Depth camera, 133: Light source, 134: Eye tracking camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 170a: Work master data, 170b: Base point master data, 170c: Body size master data, 170d: Tool master data, 170e: Fastening point master data, 170f: History data, 200: Article, 201-204: Fastening point, 210: Marker, 215: Screw, 251: Wrench, 251a: Marker, 251b: Head, 252: Extension bar, 253: Socket, 261: Left hand, 261a: Virtual object, 262: Right hand, 262a: Virtual object, 301, 301a, 301b, 311, 321a, 321b, 331 to 334, 341, 342: Virtual object, 351 to 353: Alert, 351a, 351b: Message, M1: Processing method, M2: Confirmation method, RD1, RD2: Rotation direction, c1, c2: Center position, d1 to d3: Distance

Claims

1. A display device that displays a virtual object superimposed on a real space, obtaining the position and orientation of the display device; displaying a first virtual object indicating a workable range set using the position and the direction; A work position is set at a predetermined position relative to a fastening point of an article existing in the real space; a display device that determines whether the work position is within the workable range, and outputs a first alert if the work position is outside the workable range.

2. Inferring whether work has been performed on the fastening point; The display device according to claim 1 , wherein when it is estimated that the work has been performed, a record of the work is linked to the data of the fastening location.

3. Identifying the fastening points to be worked on based on the positions of the plurality of fastening points registered in advance and the records for each of the plurality of fastening points; The display device according to claim 2 , wherein the work position is set at the predetermined position relative to the identified fastening location.

4. Receives torque values ​​from the tool that turns the screw, The display device according to claim 2 , wherein the torque value is linked to the data of the fastening point as the record.

5. After the work, it is determined whether the received torque value is equal to or greater than a predetermined torque value, and whether the number of times the screw has been tightened at the fastening point matches a predetermined number of times; The display device according to claim 4 , wherein a second alert is output when the received torque value is less than the predetermined torque value or when the number of screw tightenings is less than the predetermined number of times.

6. The display device according to any one of claims 2 to 5, wherein when the work is performed with the work position outside the workable range, first data is linked to the data of the fastening location.

7. displaying a second virtual object in correspondence with the fastening point; 6. The display device according to claim 2, wherein execution of work on the fastening location is estimated based on contact between a predetermined object and the second virtual object.

8. Measure the hand position from the image, estimating a center of rotation position of a tool for turning a screw from the plurality of positions of the hand; 6. The display device according to claim 2, wherein the execution of work on the fastening location is estimated based on the estimated center position.

9. displaying a third virtual object including information about the work to be performed on the fastening location; A display device described in any one of claims 1 to 4, wherein the information includes one or more selected from a specified torque value required to tighten the screw into the fastening point, a torque value detected by a tool used for tightening the screw, and the number of times the screw is tightened into the fastening point.

10. Repeating the acquisition of the position and the orientation of the display device and the setting of the workable range; 5. The display device according to claim 1, wherein after the first alert is output, if it is determined that the work position is within the workable range, the first alert is stopped.

11. 5. The display device according to claim 1, wherein a length of the workable range of the display device in the direction is longer than a length of the workable range in an orthogonal direction perpendicular to the direction.

12. A registration mode for registering the workable range can be executed, A display device according to any one of claims 1 to 4, wherein in the registration mode, multiple positions of the hand are measured, the position of the display device is measured, and the workable range is registered based on the distance from the position of the display device to the hand.

13. 5. The display device according to claim 1, wherein the workable range is set using the position and the orientation of the display device and physique data of a wearer of the display device.

14. An acquisition system for acquiring the physique data referenced by the display device according to claim 13, An acquisition system that acquires the physique data from a distance between a position of the display device and a position of the wearer's hand or from an image of the wearer's body.

15. A display device that displays virtual objects overlaid on real space. obtaining a position and orientation of the display device; displaying a first virtual object indicating a workable range set using the position and the direction; A work position is set at a predetermined position relative to a fastening location of an article existing in the real space; determining whether the work position is within the workable range, and outputting a first alert if the work position is outside the workable range; Processing method.

16. A program that causes the display device to execute the processing method according to claim 15.

17. A storage medium storing the program according to claim 16.

Images