Processing device, processing system, head-mounted display, processing method, program, and storage medium
The processing device and system with AR-guided HMD effectively addresses screw fastening challenges by displaying virtual objects away from hidden fastening points and integrating torque tracking, enhancing operational accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KK TOSHIBA
- Filing Date
- 2023-03-28
- Publication Date
- 2026-07-08
AI Technical Summary
Existing technologies struggle to effectively guide workers during screw fastening tasks, particularly when fastening points are difficult to see or require specific tightening forces and orders, leading to errors such as missed screws or incorrect torque application.
A processing device and system that includes a head-mounted display (HMD) with augmented reality (AR) capabilities, displaying virtual objects indicating fastening points away from their actual locations, and integrating with a tool to track torque data, ensuring accurate and ordered screw tightening.
Enhances worker guidance, reducing errors by visually guiding screw placement and torque application, ensuring efficient and correct fastening operations even when points are hidden or difficult to see.
Smart Images
Figure 0007886674000001 
Figure 0007886674000002 
Figure 0007886674000003
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to a processing device, a processing system, a head-mounted display, a processing method, a program, and a storage medium.
Background Art
[0002] There is a technology for displaying virtual objects during work to guide the work. Regarding this technology, a technology that can guide workers more effectively is required.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The problem to be solved by the present invention is to provide a processing device, a processing system, a head-mounted display, a processing method, a program, and a storage medium that can guide workers more effectively.
Means for Solving the Problems
[0005] The processing device according to the embodiment displays a virtual first object indicating the fastening portion of the screw at a position away from the fastening portion. When the processing device further receives first data from a tool that contacts a predetermined object with the first object and turns the screw, the processing device associates the first data with second data related to the fastening portion.
Brief Description of the Drawings
[0006] [Figure 1] FIG. 1 is a functional block diagram showing a processing system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing a screw tightening operation. [Figure 3] Figure 3 is a schematic diagram showing an example of output from the processing apparatus according to the embodiment. [Figure 4] Figure 4 is a schematic diagram illustrating the processing performed by the apparatus according to the embodiment. [Figure 5] Figures 5(a) and 5(b) are schematic diagrams illustrating the method of setting the base point. [Figure 6] Figures 6(a) and 6(b) are schematic diagrams showing examples of output from the processing apparatus according to the embodiment. [Figure 7] Figures 7(a) and 7(b) are schematic diagrams showing examples of output from the processing apparatus according to the embodiment. [Figure 8] Figures 8(a) and 8(b) are schematic diagrams showing examples of output from the processing apparatus according to the embodiment. [Figure 9] Figure 9 is a schematic diagram illustrating another screw-tightening operation. [Figure 10] Figure 10 is a schematic diagram showing an example of output from the processing apparatus according to the embodiment. [Figure 11] Figure 11 is a flowchart showing the processing method according to an embodiment. [Figure 12] Figure 12 is a schematic diagram illustrating a head-mounted display according to an embodiment. [Figure 13] Figure 13 is a schematic diagram illustrating the screw tightening process. [Figure 14] Figure 14 is a schematic diagram showing an example of output from a processing apparatus according to a modified embodiment. [Figure 15] Figure 15 is a schematic diagram showing the hardware configuration. [Modes for carrying out the invention]
[0007] Each embodiment of the present invention will be described below with reference to the drawings. Drawings are schematic or conceptual, and the relationships between the thickness and width of each part, as well as the ratios of the sizes of different parts, are not necessarily identical to those of reality. Even when representing the same part, the dimensions and ratios may be depicted differently in different drawings. In this specification and in each figure, elements similar to those already described are denoted by the same reference numerals, and detailed explanations are omitted as appropriate.
[0008] In the manufacturing of products, screws are sometimes used to fasten parts together. When fastening screws, the fastening points may be difficult to see, the required tightening force (tightening torque) may differ for each screw, and there may be a prescribed fastening order. As a result, errors such as forgetting to tighten screws, insufficient tightening torque, or incorrect fastening order may occur. The invention according to this embodiment is applicable to screw fastening work and can be used to avoid these errors.
[0009] Figure 1 is a functional block diagram showing a processing system according to an embodiment. As shown in Figure 1, the processing system 1 according to this embodiment includes a processing device 10, a display device 20, an imaging device 30, a tool 40, a storage device 50, and an input device 60.
[0010] The processing unit 10 processes various data. The processing unit 10 also controls the display on the display device 20. The display device 20 displays information to the user. The imaging device 30 captures images of the work in progress and repeatedly acquires images. The imaging device 30 may also acquire video. In that case, still images are sequentially extracted from the video. It is preferable that the imaging device 30 can acquire depth (depth) in addition to two-dimensional images. An imaging device 30 for obtaining two-dimensional images and another imaging device 30 for obtaining depth may be provided.
[0011] For example, the display device 20 and the imaging device 30 are incorporated into a head-mounted display (HMD). An operator who performs screwing wears the HMD. The form and display content of the display device 20 are arbitrary. For example, the HMD includes lenses positioned in front of the operator's eyes. The display device 20 projects information regarding screwing onto a transmissive lens. The user can visually recognize the actual situation through the lens and can also visually recognize the information projected onto the lens. A non-transmissive display device 20 may be positioned in front of the operator's eyes. The display device 20 displays the video acquired by the imaging device 30 and the information regarding screwing. The user can know the actual situation from the video of the imaging device 30.
[0012] In any case, the operator can visually recognize the information regarding screwing from the display by the display device 20. Thus, a technology that enhances, expands, or fuses the real world perceived by humans through data processing is known as augmented reality (AR) or mixed reality (MR).
[0013] The form of the HMD is also arbitrary. For example, as the form of the HMD, a single-lens goggle type, a two-lens glasses type, a hat type, a helmet type that completely covers the head, etc. can be adopted. Also, in addition to the imaging device 30, the HMD may be provided with an imaging device for calculating the position of the user's eyes, the direction of the line of sight, etc. The processing device 10 calculates the user's viewpoint in the display device 20 based on the images obtained by those imaging devices. That is, the HMD may have the function of an eye tracker.
[0014] The tool 40 is a wrench, a driver, etc. used for turning a screw. The tool 40 can measure a torque value. The tool 40 transmits first data regarding fastening to the processing device 10. The processing device 10 manages data regarding screwing based on the images obtained by the imaging device 30, the data received from the tool 40, etc. The storage device 50 stores the data obtained by the processing of the processing device 10, the first data from the tool 40, the data used for display control, etc.
[0015] The input device 60 is used for the user to input data into the processing device 10. For example, the input device 60 can be a mouse, a keyboard, etc. The input device 60 can also be a microphone incorporated in the HMD. Or, virtual input devices such as a keyboard and buttons may be displayed on the display device 20. In this case, the user can input data into the processing device 10 by operating the virtual input device. Data may also be input into the processing device 10 by hand gestures based on motion detection.
[0016] FIG. 2 is a schematic diagram showing a screwing operation. FIG. 3 is a schematic diagram showing an output example by the processing device according to the embodiment. A specific example of the processing system 1 according to the embodiment will be described while referring to FIGS. 2 and 3. For example, as shown in FIG. 2, an operator W performs screwing on an article (object) 100. The operator W is wearing an HMD. The article 100 is a part (module, component) used for a product, or a work-in-progress, etc.
[0017] In the illustrated example, the article 100 is large, and the operator W stands on a platform 105 to perform screwing. The article 100 includes a cylindrical member 110 and a cylindrical member 120. A plurality of fastening locations 111 to 118 are provided on the cylindrical member 110. A "fastening location" is a location where a screw is tightened and corresponds to a location where a screw hole is provided. The cylindrical member 120 is provided with openings 121 corresponding to the fastening locations 111 to 114 and openings 125 corresponding to the fastening locations 115 to 118. The operator W fastens screws to the fastening locations 111 to 114 through the openings 121 and fastens screws to the fastening locations 115 to 118 through the openings 125, respectively.
[0018] Worker W fastens screws to fastening points 111-118 using a wrench 130 with an extension bar 131 attached. The wrench 130 is an example of a tool 40. The wrench 130 includes a torque sensor and is capable of measuring the torque value when tightening screws. The torque value is an example of first data. The wrench 130 transmits the measured maximum torque value to the processing unit 10. The wrench 130 may also continuously record the measured torque values and transmit the time-series data of the torque values to the processing unit 10. In addition, the wrench 130 may transmit its identification data (ID) and the time the torque value was measured to the processing unit 10 along with the torque value.
[0019] The wrench 130 may compare the measured torque value with a preset threshold. When the torque value exceeds the threshold, a fastening record indicating that the screw has been fastened, along with the time (timestamp), is recorded in the wrench 130. The wrench 130 transmits its ID, the fastening record, and the time to the processing unit 10. The fastening record is another example of the first data.
[0020] The processing device 10 displays the user interface (UI) 200 shown in Figure 3 on the display device 20 based on the image obtained by the imaging device 30. Here, an example is described in which the display device 20 projects information onto a transparent lens. The worker W can see the UI 200 on the display device 20, and can also see his own hand, fastening points 112 and 113, opening 121, wrench 130, extension bar 131, etc., through the lens.
[0021] UI200 displays virtual objects 211-214 and virtual objects 221-224. Each of objects 211-214 is an example of the first object. Each of objects 221-224 is an example of the second object. Objects 211-214 correspond to fastening points 111-114, respectively, and are displayed at a distance from fastening points 111-114. Objects 221-224 connect fastening points 111-114 to objects 211-214, respectively.
[0022] In the illustrated example, worker W can see the fastening points 112 and 113 through the opening 121. Objects 222 and 223 are not hidden by the article, and their entirety is visible except for parts hidden by the hand, arm, wrench 130, etc. On the other hand, fastening points 111 and 114 are hidden by the cylindrical member 120, and worker W cannot see the fastening points 111 and 114. In Figure 3, for illustrative purposes, the hidden and invisible fastening points 111 and 114 are shown with dashed lines. Part of object 221 is hidden by the cylindrical member 120 and is not shown. Another part of object 221 that is not hidden by the cylindrical member 120 is shown, and the orientation of this other part of object 221 indicates the location of fastening point 111. Similarly, for object 224, only a part of object 224 is shown, indicating the location of fastening point 114.
[0023] Alternatively, with respect to objects 221 and 224, the parts hidden by the cylindrical member 120 may also be displayed, and the display manner of the parts exposed from the cylindrical member 120 may differ from the display manner of the parts hidden by the cylindrical member 120. In either case, the worker W can determine the location of the fastening points 111 to 114 based on the presence and orientation of objects 221 to 224.
[0024] In the examples shown in Figures 2 and 3, worker W is tightening a screw at the fastening point 111. The processing device 10 determines from the image of the imaging device 30 whether a predetermined object has come into contact with (overlapped with) the object 211 corresponding to the fastening point 111. For example, the predetermined object is the body of worker W. The processing device 10 determines the contact between worker W's body and object 211. The predetermined object may also be a tool. The processing device 10 may also determine the contact between the wrench 130 and object 211.
[0025] As an example, the processing unit 10 estimates the skeleton of worker W from the image. By estimating the skeleton, the positions of each joint of worker W are obtained. When the position of any joint overlaps with the area where object 211 is displayed, the processing unit 10 determines that worker W's body is overlapping with object 211.
[0026] The processing unit 10 may also determine whether a specific joint has come into contact with the object 211. For example, the processing unit 10 may determine whether at least some of the joints of the hand have overlapped with the object 211. Even if joints such as the elbow or arm overlap with the object 211, the processing unit 10 will not determine that this constitutes contact with the object 211. Here, we will describe an example in which the processing unit 10 determines whether a worker's hand has come into contact with an object.
[0027] A pose estimation model can be used to estimate the skeleton. The pose estimation model is pre-trained to estimate the skeleton (pose) of a person in an image based on the image input. The processing unit 10 inputs the image from the imaging device 30 to the pose estimation model and obtains the estimation result from the pose estimation model. The pose estimation model includes a neural network. Preferably, the pose estimation model includes a convolutional neural network (CNN). OpenPose, DarkPose, or CenterNet can be used as the pose estimation model.
[0028] For example, the display position of object 211 corresponds to the position of worker W's hand when fastening a screw to fastening point 111. By displaying object 211, worker W can easily understand where to position their hand to tighten the screw. Furthermore, object 221, which connects fastening point 111 and object 211, is displayed on UI 200. Object 221 allows worker W to easily understand the direction of fastening point 111 relative to object 211, and which way to point the wrench 130 relative to their hand. Even if fastening point 111 is difficult to see, the user can easily understand its presence and location.
[0029] Similarly, the display positions of objects 212 to 214 correspond to the positions of worker W's hands when tightening screws into fastening points 112 to 114. The user can easily grasp the existence and location of fastening points 112 to 114.
[0030] Preferably, the processing unit 10 detects multiple joints, including the fingertips and other joints of the hand. For example, the processing unit 10 detects MP joints, PIP joints, DIP joints, CM joints, IP joints, wrist joints, fingertips, etc. If any of these come into contact with the object 211, the processing unit 10 determines that screw tightening is being performed on the fastening point 111.
[0031] When worker W's hand touches object 211, the processing unit 10 determines that a screw has been fastened to fastening point 111. At the same time, the processing unit 10 receives first data regarding fastening from the wrench 130. The processing unit 10 determines that a screw has been fastened to fastening point 111 when worker W's hand touches object 211 and the first data has been received from the wrench 130. The processing unit 10 links the first data with second data regarding fastening point 111. The processing unit 10 also stores this linked data in the storage device 50. The second data is an ID indicating fastening point 111, or the location of fastening point 111, etc.
[0032] The contact of the hand with object 211 and the reception of the first data from the wrench 130 do not necessarily have to occur simultaneously. Even after the hand has made contact with object 211 and then left it, the processing unit 10 may maintain the determination that the hand is in contact with object 211 for a predetermined period of time. This allows for a more reliable linking of the first and second data, even if the worker W's hand temporarily leaves object 211, or if there is a time lag in communication between the wrench 130 and the processing unit 10. The starting point for the period during which the determination result is maintained is arbitrary. The starting point may be the moment it is first determined that the hand has made contact with object 211, or it may be the moment after contact is determined that the hand is no longer in contact with object 211.
[0033] Figure 4 is a schematic diagram illustrating the processing performed by the apparatus according to the embodiment. When the state shown in Figure 3 is captured by the imaging device 30, the processing device 10 detects the skeleton of worker W based on the image. Figure 4 is an example of the detection result. The skeleton detection reveals worker W's left elbow 301, left hand 302, right elbow 311, and right hand 312. The processing device 10 determines that screwing is being performed on the fastening point 111 when the left hand 302 or right hand 312 comes into contact with the object 211. The processing device 10 links the first data obtained during screwing on the fastening point 111 with the second data related to the fastening point 111.
[0034] For example, the processing unit 10 does not determine that screwing is being performed at the fastening point 111 even if the wrench 130, extension bar 131, left elbow 301, or right elbow 311 comes into contact with the object 211. This reduces the possibility of incorrectly determining that screwing is being performed at the fastening point 111 when the worker W's body comes into contact with the object 211 for reasons other than screwing at the fastening point 111.
[0035] If the first data includes a torque value, the processing unit 10 may determine whether fastening to the fastening point is complete. The processing unit 10 refers to a threshold value set for the fastened fastening point. If the torque value exceeds the threshold value, the processing unit 10 determines that fastening to that fastening point is complete. If the first data includes a record of screw fastening, the processing unit 10 determines that fastening to the fastening point is complete upon receiving the first data.
[0036] The processing unit 10 may determine whether a tool has come into contact with the first object, rather than whether the worker W's body has come into contact with the first object. When the processing unit 10 receives first data from the wrench 130 while the wrench 130 is in contact with the object 211, it may determine that a screw has been fastened to the fastening point 111. The processing unit 10 may determine both contact between the worker W's body and the object, and contact between the tool and the object. The processing unit 10 may also determine contact between any tool other than the tool and the first object. The processing unit 10 determines contact between a predetermined object and the first object.
[0037] Figures 5(a) and 5(b) are schematic diagrams illustrating the method of setting the base point. The display positions of objects 211-214 and objects 221-224 are calculated based on a base point. The processing unit 10 accepts the selection of the screw tightening to be performed and the setting of the base point. For example, an AR marker or QR code (registered trademark) is attached to a predetermined position on article 100. The processing unit 10 detects the AR marker or QR code (registered trademark) captured in the image. The processing unit 10 constructs spatial coordinates using the AR marker or QR code (registered trademark) as a base point. Alternatively, in some cases, it may not be possible to attach an AR marker or QR code (registered trademark) to an article. In that case, for example, as shown in Figure 5(a), worker W points to three base points P1-P3 with their finger. The processing unit 10 constructs spatial coordinates using the three points at the tip of the finger as base points.
[0038] As shown in Figure 5(b), an object 250 for setting the base point may be displayed. The worker W adjusts the position and orientation of the HMD so that the actual item overlaps with the displayed object 250. If the item is small, the worker W may adjust the position and orientation of the item so that it overlaps with the displayed object 250. When the processing unit 10 determines through image processing that the item overlaps with object 250, it constructs spatial coordinates using the position and orientation of the HMD at that time as the base point.
[0039] The processing unit 10 displays objects 211-214 and objects 221-224 at pre-set positions in the constructed spatial coordinate system for the selected screw tightening.
[0040] Alternatively, the processing device 10 may detect fastening locations from the image obtained by the imaging device 30 instead of using a reference point. For example, the processing device 10 can detect fastening locations in the image by comparing the obtained image with a pre-prepared template image of fastening locations. The processing device 10 then displays the first object and the second object corresponding to the detected fastening locations.
[0041] Figures 6(a), 6(b), 7(a), 7(b), 8(a), and 8(b) are schematic diagrams showing examples of output from the processing apparatus according to the embodiment. The processing unit 10 may change the display of each object depending on the content or progress of the work. For example, screws are tightened to fastening points 111 to 114 in the order of fastening point 111, fastening point 113, fastening point 112, and fastening point 114. In this case, the processing unit 10 displays only the object corresponding to the fastening point to be fastened at that time. When it is determined that fastening to the fastening point indicated by the object is complete, the processing unit 10 displays only the object corresponding to the next fastening point to be fastened.
[0042] In the example shown in Figure 6(a), only object 211, which corresponds to fastening point 111, is displayed. Once worker W has finished screwing the fastening point 111, only object 213, which corresponds to fastening point 113, is displayed, as shown in Figure 6(b). By displaying the objects sequentially according to the screwing order, worker W can fasten the fastening points in a predetermined order.
[0043] The appearance of the object may change depending on the number of times the screws are tightened. When multiple screws are fastened to a single component, tightening a new screw may loosen an already fastened screw. It is preferable that the already fastened screws are then tightened again. For example, worker W fastens screws at all fastening points 111 to 114, and then fastens screws at fastening points 111 to 114 again.
[0044] In the example shown in Figure 7(a), screw fastening to fastening points 111 and 113 is complete. Screw fastening to fastening points 112 and 114 is not complete. Therefore, the display of objects 211 and 213 corresponding to fastening points 111 and 113 is different from the display of objects 212 and 214 corresponding to fastening points 112 and 114.
[0045] For example, the different display modes are one or more selected from the group consisting of color, size, and shape. Here, "color" includes not only hue but also patterns such as dots and hatching, gradients, and textures. The display mode of an object may change by changing the gradient, texture, or pattern without changing the hue.
[0046] Figure 7(b) shows the state after the example shown in Figure 7(a). In this example, the screws at fastening points 111 to 114 have been tightened. The screws at fastening points 111 and 113 have been further tightened. The screws at fastening points 112 and 114 have not yet been retightened. Therefore, the display patterns of objects 211 and 213 corresponding to fastening points 111 and 113 are different from the display patterns of objects 212 and 214 corresponding to fastening points 112 and 114. Also, the display patterns of objects 211 and 213 shown in Figure 7(b) are different from the display patterns of objects 211 to 214 shown in Figure 7(a). By changing the display patterns of the objects according to the progress of the work, worker W can easily understand which fastening points need to be worked on.
[0047] As shown in Figure 7(b), the processing unit 10 may display the first data D1 received from the wrench 130 when it is determined that a specific fastening point has been tightened. In the illustrated example, the torque value is displayed as the first data. In addition, the torque value for the first tightening and the torque value for the second tightening are displayed for the specific fastening point. The fastening record may also be displayed as the first data. For example, the processing unit 10 may display a symbol (○) indicating that it has been fastened, or a symbol (×) indicating that it has not yet been fastened. The time at which the first data was received may be displayed along with each torque value or each fastening record.
[0048] The processing unit 10 may display symbols indicating the screw tightening sequence on objects 211 to 214, as shown in Figure 8(a). In the illustrated example, numbers are displayed as symbols. The symbols may be displayed superimposed on each object, or displayed near the object without superimposing. Once screw tightening to the fastening points 111 is complete, the processing unit 10 may change the symbols indicating the sequence, as shown in Figure 8(b). The processing unit 10 may also hide the object 211 corresponding to the fastening point 111 for which screw tightening has been completed.
[0049] A mark may be placed on a fastened screw to indicate completion of the work. Worker W places a mark on the screw or its vicinity once the screw tightening is complete. The screw tightening tool may automatically place the mark upon completion. The processing unit 10 may detect the mark from an image. When the processing unit 10 determines that the screw tightening is complete, it refers to the color of the mark used in that screw tightening. The processing unit 10 determines whether the number of pixels of the mark color in the image obtained by the imaging device 30 exceeds a preset threshold. If the number of pixels exceeds the threshold, the processing unit 10 determines that the screw has been marked. For example, the processing unit 10 further associates the detection result indicating that a mark has been detected with the fastening location and the first data.
[0050] Figure 9 is a schematic diagram showing another screw tightening operation. Figure 10 is a schematic diagram showing an example of output from the processing apparatus according to the embodiment. In the example shown in Figure 9, worker W is screwing in a fastening point 129 located inside the cylindrical member 120. The fastening point 129 is not visible to worker W. The cylindrical member 110 is mounted on a base 101. Worker W has positioned a wrench 130 between the base 101 and the cylindrical member 120. Worker W then inserts an extension bar 131 from below the cylindrical member 120 into the gap between the cylindrical member 110 and the cylindrical member 120 and screws in the fastening point 129.
[0051] Figure 10 shows an example of output from the display device 20 when performing the operation shown in Figure 9. In the illustrated example, the base 101 prevents worker W from reaching the position directly below the fastening point 129. Worker W is gripping one end and the middle of the wrench 130 and rotating the other end of the wrench 130. As a result, object 219, which corresponds to the fastening point 129, is located near the center of the wrench 130. Object 219 is another example of the first object.
[0052] Object 219 is positioned where worker W's left hand is located. Since worker W's left hand grips the center of the wrench 130, when the other end of the wrench 130 is rotated, worker W's left hand moves in an arc. The shape of object 219 is arc-shaped, corresponding to the range of movement of worker W's left hand.
[0053] As shown in Figure 10, if the object used to determine contact with the first object may move during screw tightening, it is preferable that the length of the first object in one direction be longer than its length in another direction, corresponding to that movement. For example, this can prevent a false determination that worker W is not tightening a screw even though they are, simply because their body has moved away from the first object. This allows for a more accurate linking of the first and second data.
[0054] Object 229, which connects the fastening point 129 and object 219, is bent to indicate the orientation of the tool. Specifically, a portion of object 229 extends in one direction, indicating the position and orientation of a portion of the wrench 130. Another portion of object 229 extends in a different direction, indicating the position and orientation of the extension bar 131. Object 229 is another example of the second object. As shown in Figure 10, the shape of the second object can be appropriately changed depending on the position of the fastening point, the tool used, the position of the hand, etc.
[0055] Figure 11 is a flowchart showing the processing method according to an embodiment. The storage device 50 stores master data 51 and history data 52. The master data 51 includes work master data 51a, tool master data 51b, base point master data 51c, and fastening location master data 51d. The master data 51 is prepared in advance before screw tightening.
[0056] In the processing method M according to the embodiment, the processing device 10 accepts the selection of a task and a tool (step S1). The task and tool are selected by the worker W. The task to be performed and the tool to be used may be instructed by a higher-level system, and the processing device 10 may accept the selection based on the instruction. Based on data obtained from the imaging device 30 or other sensors, the processing device 10 may determine the task to be performed and the tool to be used, and the processing device 10 may accept the selection based on the determination.
[0057] The work master data 51a stores the work ID, work name, item ID, and item name. The processing unit 10 can accept any of the work ID, work name, item ID, or item name as a work selection. The tool master data 51b stores the tool type and tool ID. The tool type indicates the classification of the tool by structure, external shape, or performance. The processing unit 10 can accept either the tool type or tool ID as a tool selection. If there are multiple selection candidates for work or tool, the processing unit 10 may output a query asking which candidate to select.
[0058] The processing unit 10 refers to the base point master data 51c. The base point master data 51c stores the method for setting the base point for each task. The processing unit 10 obtains the method for setting the base point for the selected task and sets the base point based on the input from the worker W (step S2). As the method for setting the base point, the method described with reference to Figure 5(a) or Figure 5(b) can be applied.
[0059] The processing unit 10 refers to the fastening location master data 51d. The fastening location master data 51d stores the fastening location ID, and for each fastening location ID, it stores the fastening position, angle, extension type, torque value, number of screw tightenings, mark color, object shape, and display mode. The fastening position indicates the location of each fastening location. The angle indicates the angle of the tool or extension bar used when tightening screws at each fastening location. The extension type indicates the classification of the extension bar by structure, external shape, or performance. The torque value indicates the magnitude of the torque required when tightening screws at each fastening location. The number of screw tightenings indicates the number of screw tightenings at each fastening location. The mark color is the color of the mark indicating completion of screw tightening. The object shape indicates the shape of the object displayed corresponding to each fastening location. The display mode indicates the display mode of each object. The display mode is set for each number of screw tightenings.
[0060] The processing unit 10 displays the first object and the second object based on the set base point and referenced data (step S3). The positions in which the first object and the second object are displayed are calculated based on the base point, fastening position, angle, extension type, tool type, etc.
[0061] For example, the extension bar length is determined by the extension type. The first object is displayed at a distance equal to the length of the extension bar, at a set angle relative to the fastening position. The second object is displayed at a set angle relative to the fastening position, within the area equal to the length of the extension bar.
[0062] If an extension bar is not used, the tool type determines the tool length. A first object is displayed at a predetermined angle relative to the fastening position, at a distance of a predetermined percentage of the tool length. This predetermined percentage is set according to the position where an operator typically grips the tool. A second object is displayed at a predetermined angle relative to the fastening position, in an area corresponding to the predetermined percentage of the tool length.
[0063] During screw tightening, the worker may not be able to grip the end of the extension bar due to obstacles or other factors. In this case, the display position of the first object may be calculated using both the length of the extension bar and the length of the tool. For example, the angle of the tool relative to the extension is determined by the extension type and the tool type. The processing unit 10 calculates the connection position between the extension bar and the tool as a position separated by the length of the extension bar at a set angle with respect to the fastening position. The processing unit 10 displays the first object at a position separated from the connection position by a predetermined percentage of the tool's length at the tool's angle. The predetermined percentage is set to, for example, "0.5". That is, it is assumed that the worker grips the center of the tool. The second object is displayed in the area between the fastening position and the connection position. Furthermore, the second object is also displayed in an area separated from the connection position by a predetermined percentage of the tool's length at the tool's angle.
[0064] The processing unit 10 repeatedly determines whether a predetermined object has come into contact with the first object (step S4). When contact with the first object is determined, the processing unit 10 determines whether it has received first data related to fastening from the tool 40 (step S5). If the first data is not received, the processing unit 10 performs step S4 again. As described above, once contact with the first object is determined, the determination result may be maintained for a predetermined time. In that case, the processing unit 10 repeats step S5 while the determination result is maintained.
[0065] When it is determined that a predetermined object has come into contact with the first object and that the first data has been received, the processing device 10 links the second data relating to the fastening location with the first data and records these data in the history data 52 (step S6). In the illustrated example, the torque value is recorded as the first data, and the fastening location ID is recorded as the second data. In addition, the work ID, tool ID, and tool type are recorded in association with the first and second data.
[0066] The processing unit 10 detects a mark of the color set in the fastening location master data 51d from the image acquired by the imaging device 30 (step S7). The processing unit 10 further associates the mark detection result with the history data recorded in step S6 and records it. The processing unit 10 determines whether the operation selected in step S1 has been completed (step S8). If the operation has not been completed, the display of the object in step S3 is repeated.
[0067] During the process, the worker can also move the base point. The worker inputs an instruction to move the base point to the processing unit 10. The instruction may be input using the input device 60, or it may be input by a hand gesture indicating the instruction to move the base point. When the processing unit 10 receives the input, it interrupts the process it is executing and proceeds to step S2. The worker resets the base point using one of the methods described above. As a result, the first object and the second object are displayed based on the reset base point.
[0068] Figure 12 is a schematic diagram illustrating a head-mounted display according to an embodiment. An embodiment may use, for example, the HMD400 shown in Figure 12. The HMD400 includes a processing unit 10, a frame 401, a projection device 421, a projection device 422, an image camera 431, a depth camera 432, a lens 441, a lens 442, a sensor 450, and a battery 460. The HMD400 is a mixed reality device.
[0069] The HMD400 is a dual-lens system, with two lenses 441 and 442 fitted into the frame 401. The worker can view the real world through lenses 441 and 442. Projection devices 421 and 422 project information onto lenses 441 and 442, respectively. For example, projection devices 421 and 422 display a first object and a second object on lenses 441 and 442. Alternatively, only one of the projection devices 421 and 422 may be provided, and information may be displayed on only one of the lenses 441 and 442.
[0070] The image camera 431 detects visible light and obtains a two-dimensional image. The depth camera 432 emits infrared light and obtains a depth image based on the reflected infrared light. The sensor 450 is a 6-axis detection sensor capable of detecting angular velocity in 3 axes and acceleration in 3 axes. The processing unit 10 detects movement of the field of view based on the detection results from the sensor 450. The battery 460 supplies the necessary power to each element of the HMD 400.
[0071] The processing unit 10 can communicate with an external storage device 50. The processing unit 10 uses data obtained from the image camera 431 and depth camera 432, data from the storage device 50, etc., to perform the processing shown in Figure 11.
[0072] The advantages of the embodiment will be explained. Traditionally, technologies such as AR or MR have been applied to screw tightening operations. One example is a technology that detects screw holes from acquired images and displays objects within those holes. Displaying objects within the screw holes can guide workers during screw tightening, making the process more efficient. Furthermore, in this technology, when screw insertion into a screw hole is detected, data related to that screw hole is linked to data obtained from the tool.
[0073] On the other hand, when the item to be screwed is large, or when the fastening point is hidden, workers may be unsure where to position their hands or tools to perform the work. In some cases, screwing may be difficult unless the tools are positioned in a specific location. To address this issue, the processing device 10 according to this embodiment displays a virtual first object indicating the fastening point not at the fastening point itself, but at a location away from the fastening point. The first object is displayed at the location where the worker's hands or tools are positioned during screwing. By referring to the first object, the worker can easily understand where to position the objects. According to this embodiment, the worker can be guided more effectively, and the worker can perform the screwing work more efficiently.
[0074] Preferably, the processing device 10 displays a virtual second object in addition to the first object. The second object connects the fastening point to the first object. The display of the second object makes it easy for the worker to determine which direction to point the tool after placing the object at the location of the first object.
[0075] In the display by the processing device 10, if the fastening point is hidden by the article, a portion of the second object exposed from the article is displayed, while another portion of the second object hidden by the article is not displayed. By hiding the second object according to its position relative to the article, it becomes easier for workers to understand the location of the fastening point on the article. Alternatively, the display manner of a portion of the second object that is not hidden by the article may differ from the display manner of another portion of the second object hidden by the article. In this case as well, it becomes easier for workers to understand the location of the fastening point on the article.
[0076] When fastening screws onto an object, some fastening points may be invisible to the worker (imaging device). In this case, the fastening points cannot be detected from the image obtained by the imaging device. If the fastening points cannot be detected, the objects corresponding to those fastening points cannot be displayed. For this reason, it is preferable that the display position of the objects be calculated based on a base point rather than based on the results of image processing. By using a base point, even if the fastening points are not visible in the image, the first and second objects can be displayed corresponding to the fastening points.
[0077] Furthermore, when detecting fastening points from an image, detection can be difficult if there are no easily identifiable elements such as marks or barcodes at the fastening points. On the other hand, from the perspective of product quality control and standards, it may not be possible to add easily identifiable elements. When using a reference point, easily identifiable elements are not required, and even for objects that are difficult to detect from an image, the first and second objects corresponding to those objects can be displayed.
[0078] Preferably, the first object is displayed at the position where the worker's hand is positioned. The processing unit 10 determines contact between the first object and the worker's hand. In typical screw tightening, the worker holds a tool in their hand and manipulates the tool with their hand to tighten the screw. By indicating the position of the hand handling the tool with the first object, the worker can be guided more effectively.
[0079] (modified version) Figure 13 is a schematic diagram illustrating a screw tightening operation. Figure 14 is a schematic diagram showing an example of output from a processing apparatus according to a modified embodiment. In the example described above, an HMD is used as the display device 20 and the imaging device 30. The display device 20 and the imaging device 30 may be implemented using something other than an HMD. For example, as shown in Figure 13, the imaging device 30 captures images of the worker W and the item 100 from the side.
[0080] The display device 20 is located near worker W. The processing device 10 displays the UI400 shown in Figure 14 on the display device 20. The UI400 displays the image (video) captured by the imaging device 30, as well as objects 211-214 and objects 221-224. Worker W tightens the screws at fastening points 111-114 while checking the UI400.
[0081] In the modified configuration, worker W can perform the screw-tightening task while viewing the information displayed on the display device 20. This allows for more effective guidance of worker W. However, to further improve worker W's work efficiency, it is preferable to use an HMD equipped with a display device 20 and an imaging device 30, as shown in Figure 12.
[0082] The above describes examples of screws being fastened to various fastening points. Embodiments of the present invention are applicable not only when screws are tightened to fastening points, but also when screws are loosened. For example, screws are loosened when performing maintenance, inspection, or repair of a product. According to embodiments of the present invention, when loosening a screw, a virtual first object indicating the fastening point is displayed. This makes it easy for the worker to understand where to place the object when loosening the screw. According to the embodiments, the worker can be guided more effectively and can perform the screw-loosening work more efficiently.
[0083] When loosening a screw, the processing unit 10 displays each object based on a reference point, similar to the example described above. The processing unit 10 also detects contact between each object and the worker's hand, or contact between each object and the tool. While loosening the screw, the processing unit 10 receives first data (e.g., torque value) from the tool turning the screw. When the processing unit 10 receives first data from the tool, it links the first data with second data related to the fastening location.
[0084] Furthermore, similar to the example described above, the processing device 10 may sequentially display the first object at each fastening location in accordance with the order in which the screws are loosened. The processing device 10 may also differentiate the display mode of the first object corresponding to the fastening location where the screw has been loosened from the display mode of the first object corresponding to the fastening location where the screw has not been loosened.
[0085] Figure 15 is a schematic diagram showing the hardware configuration. As the processing unit 10, for example, the computer 90 shown in Figure 15 is used. The computer 90 includes a CPU 91, ROM 92, RAM 93, storage device 94, input interface 95, output interface 96, and communication interface 97.
[0086] ROM92 stores programs that control the operation of computer 90. ROM92 contains the programs necessary for computer 90 to perform each of the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are loaded.
[0087] The CPU 91 includes processing circuits. The CPU 91 uses the RAM 93 as work memory and executes programs stored in at least one of the ROM 92 or the storage device 94. The storage device 94 may be used as the storage device 50 shown in Figure 1. During program execution, the CPU 91 controls each component via the system bus 98 and performs various processes.
[0088] The storage device 94 stores data necessary for program execution and data obtained through program execution. The storage device 94 includes a Solid State Drive (SSD), etc. The storage device 94 may also be used as the storage device 50.
[0089] The input interface (I / F) 95 allows the computer 90 to be connected to the input device 60. The CPU 91 can read various data from the input device 60 via the input I / F 95.
[0090] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can send data to the display device 20 via the output I / F 96 and display information on the display device 20.
[0091] The communication interface (I / F) 97 allows connection between the computer 90 and external devices. For example, the communication I / F 97 connects the tool 40 and the computer 90 via Bluetooth® communication.
[0092] The data processing performed by the processing unit 10 may be carried out by only one computer 90. Part of the data processing may also be performed by a server or other device via the communication interface 97.
[0093] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.
[0094] For example, information recorded on a recording medium can be read by a computer (or embedded system). The recording format (storage format) of the recording medium is arbitrary. For example, a computer reads a program from the recording medium and has the CPU execute the instructions written in the program based on this program. In a computer, program acquisition (or reading) may be performed via a network.
[0095] The embodiments include, for example, the following aspects: (Note 1) A virtual first object indicating the screw fastening location is displayed at a position away from the fastening location. A processing device that links the first data with second data relating to the fastening location when a predetermined object comes into contact with the first object and first data is received from a tool for turning the screw. (Note 2) The processing apparatus described in Appendix 1, which further displays a virtual second object connecting the fastening point and the first object. (Note 3) The processing apparatus according to Appendix 2, wherein, when the fastening location is hidden by an article, a portion of the second object exposed from the article is displayed, and another portion of the second object hidden by the article is not displayed, or the display manner of the portion of the second object is different from the display manner of the other portion of the second object. (Note 4) The processing apparatus according to any one of the appendices 1 to 3, which sequentially displays the first object at multiple fastening locations. (Note 5) The first object is displayed for each of the multiple fastening locations, The processing apparatus according to any one of the appendices 1 to 3, which changes the display mode of the corresponding first object according to the number of times each fastening point is fastened. (Note 6) A processing device according to any one of the appendices 1 to 5, which accepts the setting of a base point and displays the first object based on the position of the base point. (Note 7) The resetting of the aforementioned base point is possible. The processing device described in Appendix 6, which, when the base point is reset, displays the first object based on the position of the reset base point. (Note 8) A virtual first object indicating the screw fastening location is displayed at a position away from the fastening location. A processing device that displays a virtual second object connecting the fastening point and the first object. (Note 9) The processing apparatus described in any one of the appendices 1 to 8, A display device on which the above-mentioned first object is displayed, A processing system equipped with [the following features]. (Note 10) The processing apparatus described in any one of the appendices 1 to 8, A display device on which the above-mentioned first object is displayed, An imaging device for imaging an article including the fastening location, A head-mounted display equipped with [a specific feature / ability].
[0096] According to the embodiments described above, a processing device, a processing system, a head-mounted display, a processing method, a program, and a storage medium are provided that can guide workers more effectively.
[0097] Although several embodiments of the present invention have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other. [Explanation of symbols]
[0098] 1: Processing system, 10: Processing device, 20: Display device, 30: Imaging device, 40: Tool, 50: Storage device, 51: Master data, 51a: Work master data, 51b: Tool master data, 51c: Base point master data, 51d: Fastening point master data, 52: History data, 60: Input device, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: Item, 101: Base, 105: Stand, 110: Cylindrical member, 111~118: Fastening point, 120: Cylindrical member, 121: Opening, 125: Opening, 129: Fastening point, 130: Wrench, 131: Extension bar, 211-214, 219: First object, 221-224, 229: Second object, 250: Object, 301: Left elbow, 302: Left hand, 311: Right elbow, 312: Right hand, 400: HMD, 401: Frame, 421: Projection device, 422: Projection device, 431: Image camera, 432: Depth camera, 441, 442: Lens, 450: Sensor, 460: Battery, M: Processing method, W: Worker
Claims
1. A virtual first object indicating the screw fastening location is displayed at a position away from the fastening location. A processing device that links the first data with second data relating to the fastening location when a predetermined object comes into contact with the first object and first data is received from a tool for turning the screw.
2. The processing apparatus according to claim 1, further displaying a virtual second object connecting the fastening point and the first object.
3. The processing apparatus according to claim 2, wherein, when the fastening location is hidden by an article, a portion of the second object exposed from the article is displayed, and another portion of the second object hidden by the article is not displayed, or the display manner of the portion of the second object is different from the display manner of the other portion of the second object.
4. The processing apparatus according to claim 1, which sequentially displays the first object at a plurality of fastening locations.
5. The first object is displayed for each of the multiple fastening locations. The processing apparatus according to claim 1, which changes the display mode of the corresponding first object according to the number of times each fastening point is fastened.
6. The processing apparatus according to claim 1, which accepts the setting of a base point and displays the first object based on the position of the base point.
7. The resetting of the aforementioned base point is possible. The processing apparatus according to claim 6, wherein, when the base point is reset, the first object is displayed based on the position of the reset base point.
8. A virtual first object indicating the screw fastening location is displayed at a position away from the fastening location. A processing device that displays a virtual second object connecting the fastening point and the first object.
9. The processing apparatus according to any one of claims 1 to 8, A display device on which the above-mentioned first object is displayed, A processing system equipped with [the following features].
10. The processing apparatus according to any one of claims 1 to 8, A display device on which the above-mentioned first object is displayed, An imaging device for imaging an article including the fastening location, A head-mounted display equipped with [a specific feature / ability].
11. A virtual first object indicating the screw fastening location is displayed at a position away from the fastening location. A processing method for linking the first data with second data relating to the fastening location when a predetermined object comes into contact with the first object and first data is received from a tool for turning the screw.
12. A program that causes a computer to execute the processing method described in claim 11.
13. A storage medium storing the program described in claim 12.