Processing device, learning device, mixed reality device, processing method, program, and storage medium

Abstract

The present invention addresses the problem of providing a processing device, a mixed reality device, a processing method, a program, and a storage medium capable of associating data relating to a work position at which a screw is screwed with time-series data detected by a tool. Also provided is a learning device capable of causing the determiner to learn using the associated data. [Solution] A processing device according to an embodiment uses a first image to estimate a first work position at which a screw is screwed. The processing device extracts first time sequence data when the screw is screwed at the first work position from time sequence data acquired by a tool for screwing the screw. The processing device associates the first time sequence data with data relating to the first job position.

Description

Technical Field

[0001] Embodiments of the present invention relate to a processing device, a learning device, a mixed reality device, a processing method, a program and a storage medium. Background Art

[0002] Among the operations for objects, there is the operation of screwing a screw. In addition, when screwing a screw, a tool capable of detecting torque may be used. A technology capable of automatically associating data related to the operation position of screwing a screw with time series data detected by the tool is required.

[0003] Prior art literature:

[0004] Patent Literature:

[0005] Patent Document 1: International Publication No. 2021 / 176645 Summary of the invention

[0006] Problems to be solved by the invention:

[0007] One problem to be solved by the present invention is to provide a processing device, a mixed reality device, a processing method, a program and a storage medium capable of establishing associations between data related to the working position (working part) of screwing a screw and time series data detected by a tool. In addition, another problem to be solved by the present invention is to provide a learning device capable of using the associated data to enable a judge to learn.

[0008] Means used to solve the problem:

[0009] The processing device according to the embodiment estimates a first working position where a screw is screwed using a first image. The processing device extracts first time series data when the screw is screwed at the first working position from time series data acquired by a tool for screwing the screw. The processing device associates the first time series data with data about the first working position. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram showing the configuration of a processing system according to an embodiment.

[0011] Figure 2 is a schematic diagram of an example image.

[0012] Figure 3 It is a schematic diagram illustrating a working situation.

[0013] Figure 4 is a graph illustrating time series data.

[0014] Figure 5 (a)~ Figure 5 (d) is a graph illustrating time series data.

[0015] Figure 6 Is a table of the instantiated database.

[0016] Figure 7 (a)~ Figure 7 (d) is a schematic diagram illustrating the types of abnormalities.

[0017] Figure 8 (a)~ Figure 8 (d) is a schematic diagram illustrating the types of abnormalities.

[0018] Fig. 9 (a)~ Fig. 9 (d) is a graph illustrating time series data.

[0019] Fig.10 It is a flowchart showing the processing method involved in the embodiment.

[0020] Fig.11 It is a flowchart showing the processing method involved in the embodiment.

[0021] Fig.12 It is a schematic diagram showing a display example displayed by the processing device involved in the embodiment.

[0022] Fig.13 It is a schematic diagram showing a display example displayed by the processing device involved in the embodiment.

[0023] Fig.14 It is a schematic diagram showing the configuration of a processing system according to a modification of the embodiment.

[0024] Fig.15 is a schematic diagram illustrating a mixed reality device.

[0025] Fig.16 It is a schematic diagram which illustrates the object of the work.

[0026] Fig.17 It is a schematic diagram used to illustrate the processing performed by the mixed reality device involved in the implementation method.

[0027] Fig.18 It is a schematic diagram used to illustrate the processing performed by the mixed reality device involved in the implementation method.

[0028] Fig.19 It is a schematic diagram used to illustrate the processing performed by the mixed reality device involved in the implementation method.

[0029] Fig. 20 It is a schematic diagram used to illustrate the processing performed by the mixed reality device involved in the implementation method.

[0030] Fig.21 It is a schematic diagram used to illustrate other processing performed by the mixed reality device involved in the implementation method.

[0031] Fig. 22 This is a diagram for explaining the method of calculating the center coordinates.

[0032] Fig.23 This is a diagram for explaining the method of calculating the center coordinates.

[0033] Fig.24 This is a diagram for explaining the method of calculating the center coordinates.

[0034] Fig.25 This is a diagram for explaining the method of calculating the center coordinates.

[0035] Fig.26 This is a diagram for explaining the method of calculating the center coordinates.

[0036] Fig. 27 This is a diagram for explaining the method of calculating the center coordinates.

[0037] Fig.28 It is a schematic diagram illustrating a working situation.

[0038] Fig.29 This is a schematic diagram showing the hardware configuration.

[0039] Description of reference numerals:

[0040] 1: Photographic device, 2: Tool, 3: Processing device, 4: Input device, 5: Display device, 6: Storage device, 8: Learning device, 10, 10a: Processing system, 50a: Work master data, 50b: Worker master data, 50c: Fastening position master data, 51: First judgment device, 52: Second judgment device, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Input output interface, 97: communication interface, 98: system bus, 100: operator, 101: wall, 102: table, 110: object, 111-114: fastening position, 120: wrench, 121: mark, 125: screw, 127: foreign matter, 128: socket, 130: extension rod, 140: object, 141: fastening position, 150: table, 171: time series data, 172: alarm, 172a: fastening position, 172b: message, 181: name, 182: job name, 183: job time, 184: judgment result, 185: judgment result, 186: time series data, 186a~186c: time series data, 187: object, 200: MR device, 201: frame, 211, 212: lens, 221, 222: projection device, 231: image camera, 232: depth camera, 240: sensor, 241: microphone, 250: processing management device, 260: battery, 270: storage device, 300: item, 301-308: fastening position, 310: mark, 351: left hand, 352: right hand, 361, 362: virtual object, 401-408: virtual object, 450: alarm, IMG: image, TD0, TD1a-TD1d, TD2a-TD2d, TD3a-TD3d, TD4a-TD4d, TDa, TDb: time series data DETAILED DESCRIPTION

[0041] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the accompanying drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the size between the parts, etc. are not necessarily the same as in reality. In addition, even when the same part is represented, it is sometimes represented in a way that the size or ratio is different from each other according to the accompanying drawings. In addition, in the specification and each figure of the present application, the same mark is added to the same element as the element already described, and the detailed description is appropriately omitted.

[0042] Figure 1 It is a schematic diagram showing the configuration of a processing system according to an embodiment.

[0043] The embodiment of the present invention can be applied to the screw tightening operation using a tool. Figure 1As shown, it includes a photographic device 1, a tool 2, a processing device 3, an input device 4, a display device 5 and a storage device 6.

[0044] The photographing device 1 photographs a worker who is working. For example, the photographing device 1 includes a camera that obtains an RGB image. The photographing device 1 may also obtain a depth image in addition to the RGB image. A plurality of photographing devices 1 may also photograph the worker.

[0045] The operator uses the tool 2 to tighten or loosen the screw relative to the object. Hereinafter, tightening the screw and loosening the screw are referred to as screwing. The object is a component, unit or semi-finished product used to make a product. The tool is a wrench.

[0046] The tool 2 can detect torque. In addition, the tool 2 can transmit the detected torque to a preset device through wireless communication. While the operator is tightening a screw with the tool 2, the tool 2 detects the torque.

[0047] The processing device 3 receives the continuous images (videos) photographed by the photographing device 1. The processing device 3 uses the images to infer the working position (working part) of the operator screwing the screw. In addition, the processing device 3 continuously receives the torque from the tool 2. Thus, time series data representing the change of the torque with respect to time is obtained.

[0048] The processing device 3 extracts data when the screw is screwed at the working position from the time series data. The processing device 3 associates the extracted time series data with data (such as ID) about the working position. The processing device 3 may also associate identification information (name or ID) of the operator, time, identification information of the tool, etc. with the data about the working position.

[0049] The input device 4 is used by the operator to input information to the processing device 3. For example, the input device 4 includes a keyboard, a mouse, or a touch pad. The input device 4 may also include a microphone. The operator can input information to the processing device 3 by speaking to the input device 4. In addition to the input device 4, the operator can also input information to the processing device 3 through gestures, etc.

[0050] The display device 5 displays information to the operator. The display device 5 includes a monitor or a projector. For example, the processing device 3 causes the display device 5 to display information for supporting the tightening work. Hereinafter, the case where the processing device causes the display device to display information is also referred to as "the processing device displays information".

[0051] The storage device 6 stores data required for processing by the processing device 3, data obtained by processing by the processing device 3, and the like. For example, the storage device 6 stores data on tools used in work, data used when estimating work positions, and data associated with each other by the processing device 3, and the like.

[0052] Figure 2 is a schematic diagram of an example image. Figure 3 It is a schematic diagram illustrating a working situation.

[0053] Figure 2 This is an example of an image obtained by photographing a worker who is working. Figure 2 In the image IMG shown, a worker 100 is shown. The worker 100 stands in front of a wall 101 and is working on an object 110 . The object 110 is placed on a table 102 .

[0054] As an example, the operator 100 holds the wrench 120 in his right hand and tightens screws at the tightening positions 111 to 114 of the article 110. The operator 100 tightens the screws in the order of the tightening position 111, the tightening position 112, the tightening position 113, and the tightening position 114. If the tightening of the screws at the tightening positions 111 to 114 is considered as one group, four groups of screw tightening are performed.

[0055] During the operation, the processing device 3 infers at which fastening position the screw is being screwed. Specifically, the processing device 3 performs hand tracking on the image IMG and measures the coordinates of the left hand and the right hand of the operator 100. Hereinafter, without specifically distinguishing between the left hand and the right hand, at least one of the left hand and the right hand will be referred to as "hand". The hand includes multiple joints such as the DIP joint, the PIP joint, the MP joint, and the CM joint. The coordinates of one of the above joints are used as the coordinates of the hand. The center of gravity positions of multiple joints can also be used as the coordinates of the hand. Alternatively, the center coordinates of the entire hand can also be used as the coordinates of the hand.

[0056] When tightening the screw with a wrench 120, Figure 3As shown, one hand holds the head of the wrench 120, and the other hand holds the handle of the wrench 120. Therefore, during the operation, the hand holding the head is close to one of the fastening positions 111 to 114. The coordinates of the fastening positions 111 to 114 are pre-registered in the storage device 6. The processing device 3 calculates the distance between each hand of the operator 100 and each fastening position 111 to 114. When the distance between one of the hands of the operator 100 and one of the fastening positions 111 to 114 is lower than a preset threshold, the processing device 3 infers that a screw is being screwed into one of the fastening positions 111 to 114. Hereinafter, a fastening position among one or more fastening positions at which a screw is inferred to be screwed is referred to as a "working position".

[0057] An inference model for inferring the work position from the image may also be prepared. The inference model is pre-learned to output an inference result of the work position in accordance with the input of the image. The inference model preferably includes a neural network (particularly a convolutional neural network). The processing device 3 inputs the image to the inference model and obtains an inference result of the work position from the inference model.

[0058] Figure 4 and Figure 5 (a)~ Figure 5 (d) is a graph illustrating time series data.

[0059] During operation, the wrench 120 obtains, for example, Figure 4 The time series data shown. Figure 4 In FIG. 1 , the solid line represents the torque detected by the wrench 120. The dotted line represents the rotation angle detected by the wrench 120. The horizontal axis represents time, and the vertical axis represents the magnitude of the torque and the magnitude of the rotation angle.

[0060] If the working position is estimated, the processing device 3 uses the shooting time of the image used in the estimation as the time to screw the screw at the working position. The processing device 3 refers to the time when each torque is detected in the time series data. The processing device 3 extracts the data when the screw is screwed at the working position from the time series data based on the time when the work is performed and the time when each torque is detected.

[0061] For example, in the overall time series data TD0, the periods for tightening the screws at the fastening positions (fastening parts) 111 to 114 are determined. The processing device 3 extracts the time series data TD1a to TD1d, the time series data TD2a to TD2d, the time series data TD3a to TD3d, and the time series data TD4a to TD4d of each period from the time series data TD0. The time series data TD1a, TD2a, TD3a, and TD4a respectively represent the torques detected when the screws are tightened at the fastening position 111 for the first, second, third, and fourth times. The same is true for other time series data. For example, the time series data TD1b, TD2b, TD3b, and TD4b respectively represent the torques detected when the screws are tightened at the fastening position 112 for the first, second, third, and fourth times.

[0062] Processing device 3 such as Figure 5 (a)~ Figure 5 As shown in (d), the time series data of the period of screwing the screw at each fastening position is extracted from the overall time series data TD0. In the example shown in the figure, the time series data TD1a, TD1b, TD1c and TD1d are extracted. The processing device 3 associates the data related to the fastening positions 111 to 114 with the time series data TD1a to TD1d respectively.

[0063] As an example, based on one or more images (first images) taken at a certain time, it is inferred that a screw is being screwed into the fastening position 111. The processing device 3 extracts the time series data TD1a (first time series data) when the screw is being screwed into the fastening position 111 (first working position) from the time series data TD0. Based on other images (second images) taken after the first image, it is inferred that a screw is being screwed into the fastening position 112. The processing device 3 extracts the time series data TD1b (second time series data) when the screw is being screwed into the fastening position 112 (second working position) from the time series data TD0.

[0064] exist Figure 2 and Figure 4 In the example shown, four sets of screw tightening are repeated for the tightening positions 111 to 114. Therefore, the data after the time series data TD1a to TD1d also specifies the period during which the screws are tightened for the tightening positions 111 to 114. As a result, the processing device 3 extracts the time series data TD2a to TD2d, the time series data TD3a to TD3d, and the time series data TD4a to TD4d for each period from the time series data TD0.

[0065] Figure 6 Is a table of the instantiated database.

[0066] For example Figure 6 As shown, the processing device 3 establishes associations between the various data. Figure 6 The table 150 shown includes operator ID 151, time 152, data name 153, operator name 154, tool ID 155, fastening position ID 156, torque 157, proficiency 158 and number of times 159. Operator ID 151 is identification information of the operator who performed the operation. Time 152 is the time when the operation was performed, based on the time when the image was photographed and the time when the torque was detected. Data name 153 is the name of the extracted time series data. Operator name 154 is the name of the operator who performed the operation. Tool ID 155 is identification information of the tool used in the operation. Fastening position ID 156 is identification information of the fastening position for tightening the screw. Torque 157 is the maximum value of the torque detected when tightening the screw at the fastening position. Proficiency 158 indicates the proficiency of the operator. Number of times 159 is the number of times the screw was tightened at the fastening position. Figure 2 and Figure 5 In the example shown, the screw is tightened four times at one tightening position. Therefore, any numerical value from "1" to "4" is registered in the number of times 159. The number of times is counted by the tool 2 or the processing device 3.

[0067] Advantages of the embodiment will be described.

[0068] The time series data of the tool can be used for various purposes. For example, based on the waveform of the time series data, it is possible to determine whether the operation is normal. Or, based on the waveform of the time series data, it is possible to determine whether the operation is performed efficiently. On the other hand, the torque is continuously detected by the tool during the operation. Therefore, even if the time series data obtained by the tool includes an abnormal waveform, it is impossible to determine at which position the abnormal waveform is obtained when the screw is screwed. Therefore, it is difficult to use time series data for verification.

[0069] In an embodiment of the present invention, a first image showing a situation of an existing operation is used to infer the operation position of screwing a screw. In addition, time series data is obtained from a tool for screwing a screw. Then, data when the screw is screwed at the operation position is extracted from the time series data. Thus, the extracted time series data can be associated with data about the operation position. As a result, it is possible to determine for which fastening position each part of the time series data detected by the tool is data when the screw is screwed.

[0070] Whether the extracted time series data is normal can also be judged as described above. For example, the processing device 3 judges whether the torque exceeds a preset first threshold value in the extracted time series data. In addition, the processing device 3 judges whether the period (length of time) of the extracted time series data exceeds a preset second threshold value. The second threshold value for the period is different from the first threshold value for the torque. When the torque is greater than the first threshold value, or when the period of the time series data is longer than the second threshold value, the processing device 3 judges that the time series data is abnormal.

[0071] For example, about Figure 4 In the time series data shown, the period of time series data TD3b is longer than that of time series data TD3a, TD3c or TD3d. Similarly, the period of time series data TD4b is longer than that of time series data TD4a, TD4c or TD4d. Therefore, time series data TD3b and time series data TD4b are judged to be abnormal. The other time series data are judged to be normal.

[0072] When judging whether the time series data is normal, the first judger learned in advance can also be used. The first judger outputs a judgment result indicating whether the time series data is normal in accordance with the input of the time series data. For example, the first judger classifies the time series data into a normal class or an abnormal class. The first judger outputs credibility for each class. The class with the highest credibility represents the classification of the time series data.

[0073] The first judger includes a neural network. Preferably, the first judger includes a recurrent neural network (RNN). In order to improve the accuracy of the judgment, the RNN more preferably includes a long short-term memory (LSTM) structure. Alternatively, the first judger may also include a convolutional neural network (CNN). In this case, the time series data is converted into an image and input to the first judger. The first judger outputs a judgment result corresponding to the shape or length of the time series data represented by the image.

[0074] Figure 7 (a)~ Figure 7 (d) and Figure 8 (a)~ Figure 8 (d) is a schematic diagram illustrating the types of abnormalities.

[0075] The first determiner may also determine the type of abnormality. The abnormality can be classified into Figure 7 (a) Figure 7 (c) Figure 8 (a) and Figure 8 There are 4 types shown in (c).

[0076] exist Figure 7 (a) and Figure 7 In the example shown in (b), the screw 125 is smaller than the threaded hole of the fastening position 111, or the threaded hole is larger than the screw 125. Even if the screw 125 is tightened to the end, the front end of the screw 125 cannot reach the bottom of the threaded hole. Figure 7 As shown in (a), in the first half of screw tightening, the torque is smaller than usual. Then, when the head of the screw 125 contacts the tightening position, the torque increases sharply. In this abnormal type, the tightening is insufficient and the screw is easy to loosen.

[0077] exist Figure 7 (c) and Figure 7 In the example shown in (d) of FIG. 1 , the screw 125 is inserted obliquely relative to the threaded hole. In this case, a greater force than usual is required to tighten the screw 125. Figure 7 As shown in (c), a torque larger than normal is detected. If the screw is tightened continuously in this state, excessive force is applied to the fastening position 111, and the fastening position 111 may be damaged.

[0078] exist Figure 8 (a) and Figure 8 In the example shown in (b), a foreign object 127 is attached to the screw 125 or the threaded hole. In this case, the tightening of the screw 125 is hindered by the foreign object 127. Figure 8 As shown in (c), a torque larger than normal is detected. If the screw is tightened continuously, excessive force is applied to the tightening position 111 or the screw 125, and the tightening position 111 or the screw 125 may be damaged.

[0079] exist Figure 8 In the example shown in (c), the screw 125 is larger than the threaded hole, or the threaded hole is smaller than the screw 125. In this case, Figure 8 As shown in (d), from the first half of screw tightening, the torque is larger than usual. Then, the torque gradually increases as the screw 125 is turned. If the screw tightening is continued, excessive force is applied to the fastening position 111, and the fastening position 111 may be damaged.

[0080] The first judgement device is trained to judge whether Figure 7 (a)~ Figure 8 The type of abnormality shown in (d) is shown in FIG. In learning, the time series data of the abnormality is used as input data, and the annotation (label) indicating the type of the abnormality is used as teaching data. By determining the type of abnormality by the first determiner, the operator can easily understand how to deal with the abnormality.

[0081] When judging whether the time series data is normal, both the comparison of the time series data with the threshold value and the judgment of the first judgement device may be performed. For example, the processing device 3 compares the torque in the extracted time series data with the threshold value, and compares the period of the time series data with the threshold value. Thus, it is judged whether the time series data is normal. If it is judged that the time series data is abnormal, the processing device 3 inputs the time series data to the first judgement device. Thus, the type of abnormality is judged by the first judgement device.

[0082] Alternatively, regardless of the judgment result obtained based on the comparison with the threshold value, the processing device 3 may input the time series data to the first judgement device. When the time series data is judged to be abnormal by either the comparison with the threshold value or the judgment of the first judgement device, the processing device 3 judges the time series data to be abnormal.

[0083] The extracted time series data can also be used to judge the proficiency of the operator. "Proficiency" expresses the skill of the operator and indicates the degree of efficiency with which the work is performed. The processing device 3 uses a second judgement device to judge the proficiency. The second judgement device judges the proficiency of the action represented by the time series data in accordance with the input of the time series data. For example, the second judgement device classifies the time series data into one of the classes of "experienced person" or "beginner". The second judgement device outputs credibility for each class. The class with the highest credibility represents the classification of the time series data. The second judgement device can also classify the time series data into one of three or more classes.

[0084] The second judgement device includes a neural network. Preferably, the second judgement device includes an RNN. In order to improve the accuracy of the judgment, the RNN preferably includes an LSTM structure. Alternatively, the second judgement device may also include a CNN. In this case, the time series data is converted into an image and input to the second judgement device. The second judgement device outputs a judgment result corresponding to the shape or length of the time series data represented by the image.

[0085] For example, the time series data when the operation includes useless movements is longer than the time series data when there is no useless movement in the operation. In addition, when the operation includes useless movements, the deviation of the time length required for each screw tightening is larger than that when there is no useless movement in the operation. When the operation includes useless movements, the deviation of the waveform of each screw tightening is also larger than that when there is no useless movement in the operation.

[0086] The second judgement device is trained in advance so as to be able to output the proficiency in accordance with the input of the time series data. In the learning, the time series data is used as the input data, and the annotation indicating the proficiency of the time series data is used as the teaching data. By judging the proficiency by the second judgement device, the operator can easily understand which part of the operation has the aspect that should be improved.

[0087] The first and second judges are prepared for each combination of the structure of the fastening position, the structure of the screw, the type of tool, the required torque, and the number of times the screw is fastened. The structure of the fastening position is the diameter of the threaded hole, the depth of the threaded hole, etc. The structure of the screw is the diameter of the screw, the shape (type) of the screw, etc. It is also possible to prepare each judge for each fastening position.

[0088] According to the embodiment of the present invention, data for each judgement device to learn can also be prepared. Figure 4 The processing of extracting a part of the time series data from the whole time series data can be used to prepare the learning data of the determination device.

[0089] Specifically, the processing device 3 extracts the time series data TD1a to TD1d from the time series data TD0, and associates the time series data TD1a to TD1d with the fastening positions 111 to 114, respectively. The time series data TD1a associated with the fastening position 111 can be used to learn the judgement device used in the operation of screwing the screw at the fastening position 111. Similarly, the time series data TD1b to TD1d associated with the fastening positions 112 to 114 can be used to learn the judgement device used in the operation of screwing the screw at the fastening positions 112 to 114. That is, through the processing of the processing device 3, the data referenced during the learning can be automatically marked for each extracted time series data. In addition, when screwing the screws at the fastening positions 111 to 114, if the tools used, the structure of the threaded holes, the required torque, etc. are the same, the judgement device used in the operation of screwing the screws at these fastening positions can also be made common.

[0090] The extracted time series data may be further labeled with other labels. For example, the processing device 3 uses a threshold value to determine whether the extracted time series data is normal. The processing device 3 assigns an annotation indicating normal or abnormal to each time series data. Thereafter, the time series data determined to be abnormal is confirmed by the operator or manager, and an annotation indicating the specific type of abnormality is assigned. Thus, data for the first determiner to learn is prepared. Even in the case where the processing device 3 determines that it is abnormal, if the operator or manager determines that the time series data is normal, the abnormal annotation is replaced with a normal annotation. The processing device 3 may also attach the proficiency of the operator who performed the operation as an annotation to the extracted time series data. As Figure 6 As shown in FIG. 1 , the proficiency level of each worker is preliminarily registered in the database. Thus, data for the second determination device to learn is prepared.

[0091] It can also be targeted at Figure 5 The time series data of each screw tightening is shown as follows, and the judgment of the first judgement device and the judgment of the second judgement device are executed. Alternatively, a plurality of time series data may be combined for each tightening position, and the judgment may be executed for the combined time series data.

[0092] Fig. 9 (a)~ Fig. 9 (d) is a graph illustrating time series data.

[0093] exist Figure 4 Among the time series data TD0 shown, the time series data TD1a, TD2a, TD3a, and TD4a are data obtained when the screw is tightened to the tightening position 111, and are associated with the tightening position 111. The processing device 3 combines the time series data associated with the common tightening position. As a result, Fig. 9 As shown in (a), the time series data TDa is obtained by combining the time series data TD1a, TD2a, TD3a and TD4a. Similarly, the processing device 3 combines the time series data obtained when the screws are tightened at the tightening positions 112 to 114 respectively. Fig. 9 (b)~ Fig. 9 As shown in (d), time series data TDb to TDd are obtained.

[0094] Fig.10 and Fig.11 It is a flowchart showing the processing method involved in the embodiment.

[0095] The storage device 6 stores Fig.10The operation master data 50a, the operator master data 50b, the fastening position master data 50c, the first judgement device 51 and the second judgement device 52 are shown. Before screw fastening, the operation master data 50a, the operator master data 50b, the fastening position master data 50c, the first judgement device 51 and the second judgement device 52 are prepared in advance.

[0096] First, the processing device 3 receives selection of a job and selection of a worker (step S0). In selecting a job, the processing device 3 refers to the job master data 50a.

[0097] In the job master data 50a, the job ID, job name, item ID, and item name are registered. The processing device 3 can accept one of the job ID, job name, item ID, and item name as the job selection. In the operator master data 50b, the operator ID, operator name, and proficiency are registered. The processing device 3 can accept the operator ID or the operator name as the operator selection.

[0098] For example, the operator selects the job and the operator. The superior system may also instruct the job to be performed, and the processing device 3 accepts the selection based on the instruction. The processing device 3 may also determine the next job to be performed based on data obtained from the photographic device 1, a sensor installed at the job site, or a reader installed at the job site. The processing device 3 selects the job based on the judgment result. The job and the operator may also be automatically selected based on a pre-prepared schedule, etc. The operator may also be pre-associated with each job. In this case, if the job is selected, the operator is also automatically selected.

[0099] If the operation and the operator are selected, the photographic device 1 starts photographing (step S1). The photographic device 1 repeatedly obtains images during the operation. In addition, if the operation starts, the processing device 3 receives the time series data of the torque from the tool 2 (step S2). The processing device 3 uses the image to infer the operation position where the screw is screwed (step S3). The processing device 3 extracts the time series data during the screwing of the screw from the time series data. The processing device 3 associates the extracted time series data with the data about the operation position (step S4). In addition, the processing device 3 also associates the operator ID, operator name, proficiency, etc. with the extracted time series data.

[0100] When estimating the working position, the processing device 3 refers to the fastening position master data 50c. In the fastening position master data 50c, the fastening position ID is stored, and the fastening position (the position of the threaded hole), the required torque value, and the required number of screw tightening times are registered for each fastening position ID. When the coordinates of the hand are used in the estimation, the processing device 3 can estimate the working position using the fastening position registered in the operator master data 50b. The processing device 3 associates the fastening position ID with the extracted time series data and saves it in the fastening position master data 50c.

[0101] If the time series data is extracted, the processing device 3 uses the first threshold and the second threshold to determine whether the time series data is normal (step S5). That is, the processing device 3 determines whether the torque included in the time series data is greater than the first threshold, and whether the period of the time series data is longer than the second threshold. In the case where the torque is greater than the first threshold, or the period is longer than the second threshold, the processing device 3 determines that the time series data is abnormal. In the case where the time series data is abnormal, the manual annotation described later is performed on the time series data (step S20).

[0102] When the time series data is determined to be normal based on the threshold, the processing device 3 combines the time series data associated with the same fastening position (step S6). Fig. 9 (a)~ Fig. 9 As shown in (d), a plurality of time series data are combined, and one time series data is obtained for each fastening position. The processing device 3 inputs the combined time series data to the first judgement device 51. The processing device 3 judges whether the time series data is normal based on the output from the first judgement device 51 (step S7). In addition, the processing device 3 inputs the combined time series data to the second judgement device 52. The processing device 3 judges the proficiency of the operator based on the output from the second judgement device 52 (step S8).

[0103] The processing device 3 displays information on the display device 5 based on the judgment result of the first judgement device 51 or the second judgement device 52 (step S9). As an example, when the time series data is judged to be abnormal, the processing device 3 displays the type of the abnormality in the operation. In the case where countermeasures are pre-registered for each type of abnormality, the processing device 3 may also display the countermeasures. By displaying information in real time during the operation, the operator can immediately realize the occurrence of the abnormality. The operation can be resumed immediately, and useless operations can be reduced. When the time series data is judged to be normal, the processing device 3 can either display information indicating that the operation is normal or not display the information. When the operation is normal, by omitting its display, the operator can easily concentrate on the operation.

[0104] As another example, when it is determined based on the time series data that the proficiency is low, the processing device 3 displays the determination result. The determination result of the proficiency may be displayed during the operation or after the operation. When the operator's proficiency is determined to be high, the processing device 3 may display information indicating that the proficiency is high or may not display the information.

[0105] The processing device 3 determines whether all the jobs are completed (step S10). If there are any jobs that have not been completed, the processing device 3 accepts the selection of a job again in step S0.

[0106] Fig.11 : is a flowchart showing the processing of manual annotation in step S20. First, the processing device 3 displays the time series data judged to be abnormal to the user (step S21). The user is a person who has sufficient knowledge to judge whether the time series data is normal. The user judges whether the displayed time series data is normal. The user uses the input device 4 to give annotations indicating normality or abnormality to the time series data. The processing device 3 accepts the assigned annotations (step S22). In addition, in the case of giving an abnormal annotation, the user uses the input device 4 to input the type of the abnormality to the processing device 3. The processing device 3 accepts the input type of the abnormality (step S23).

[0107] The processing device 3 associates the input annotation with the time series data (step S24). The processing device 3 also associates the type of abnormality with the time series data to which the abnormal annotation is assigned. The processing device 3 saves the time series data associated with the annotation as learning data for the first determiner 51 in the storage device 6 (step S25).

[0108] pass Fig.10 and Fig.11 The flowchart shown in the figure determines the normality of the time series data and the proficiency of the operator. Furthermore, through the processing of step S4, the time series data is automatically given the labels required for the learning of the first judgement device 51 and the second judgement device 52. In addition, the processing of step S20 is performed for the time series data judged to be abnormal through the processing of step S5. As a result, the labels are given with higher accuracy, and the type of abnormality is also given. As a result, data more suitable for learning can be prepared.

[0109] Fig.12 and Fig.13 It is a schematic diagram showing a display example displayed by the processing device involved in the embodiment.

[0110] The processing device 3 displays a graphical user interface (GUI) on the display device 5 based on the determination result of the first determiner 51 and the determination result of the second determiner 52 .

[0111] Fig.12 GUI 170 is shown which is displayed based on the judgment result of the first judgment device 51. In GUI 170, time series data 171 and alarm 172 are displayed. Time series data 171 is data obtained in the operation being performed. Alarm 172 includes an estimated fastening position 172a and a message 172b. Message 172b indicates the type of abnormality obtained based on the judgment result of the first judgment device 51. When the time series data is judged to be abnormal, GUI 170 is displayed during the operation. The operator can know in real time that an abnormality has occurred during the operation based on the display of GUI 170.

[0112] Fig.13 The GUI 180 displayed based on the judgment result of the first judgment device 51 and the judgment result of the second judgment device 52 is shown. In the GUI 180, the name 181 of the operator who performed the operation, the name of the operation 182, the operation time 183, the judgment result of normality 184, the judgment result of proficiency 185 and the time series data 186 are displayed. The operation time 183 is the period of the time series data of each fastening position. The judgment result 184 indicates the normality of the executed operation. In the example shown in the figure, the number of normal operations or abnormal operations among the executed operations is shown. The judgment result 184 is counted based on the judgment result of the first judgment device 51. The judgment result 185 indicates the proficiency of the executed operation based on the judgment result of the second judgment device 52. The time series data 186 indicates the time series data obtained in the operation. According to the display of the GUI 180, the operator can easily grasp the accuracy of the operation, the speed of the operation, etc. In addition, it can also be used to give feedback to the operator whether there are aspects that should be improved in the operation.

[0113] exist Fig.13 In the example shown, time series data 186a to 186c are displayed. Time series data 186a and time series data 186b are judged to be normal, and time series data 186c is judged to be abnormal. The display of time series data 186c judged to be abnormal may also be different from the display of time series data 186a and the display of time series data 186b. In the example shown in the figure, a rectangular object (object) 187 is added to the time series data 186c to emphasize the time series data 186c. As a result, the operator can easily understand which operation has an abnormality. It is also possible to replace the object with a mark, etc. The color of the time series data judged to be abnormal may also be different from the color of the time series data judged to be normal.

[0114] Fig.14 It is a schematic diagram showing the configuration of a processing system according to a modification of the embodiment.

[0115] Fig.14The processing system 10a shown is Figure 1 Compared with the processing system 10 shown in the figure, it also includes a learning device 8. In the processing system 10a, the functions of the photographing device 1, the tool 2, the processing device 3, the input device 4, the display device 5 and the storage device 6 other than the learning device 8 are substantially the same as the functions of these elements in the processing system 10.

[0116] The learning device 8 causes the first judgement device 51 and the second judgement device 52 to perform learning. In the learning, the learning data to which the annotations are automatically added by the processing device 3 and the learning data prepared by manual annotation (step S20) are used. The learning device 8 saves the learned first judgement device 51 and the learned second judgement device 52 in the storage device 6. The processing device 3 uses the first judgement device 51 and the second judgement device 52 learned by the learning device 8 to perform the judgment of the time series data.

[0117] The timing at which the learning device 8 performs learning is arbitrary. The learning device 8 may cause the first determiner 51 or the second determiner 52 to perform learning each time new learning data is prepared. The learning device 8 may cause the first determiner 51 or the second determiner 52 to perform learning at a timing when a predetermined number of learning data are saved in the storage device 6. The learning device 8 may cause the first determiner 51 or the second determiner 52 to perform learning at a timing after a predetermined period of time has passed since the last learning. The learning device 8 may be prepared separately from the processing device 3, or the processing device 3 may have the function of the learning device 8.

[0118] Fig.15 is a schematic diagram illustrating a mixed reality device.

[0119] Figure 1 The processing system 10 shown may also be implemented as a mixed reality (MR) device. Fig.15 The MR device 200 shown includes a frame 201 , a lens 211 , a lens 212 , a projection device 221 , a projection device 222 , an image camera 231 , a depth camera 232 , a sensor 240 , a microphone 241 , a processing device 250 , a battery 260 , and a storage device 270 .

[0120] The image camera 231 and the depth camera 232 are examples of the imaging device 1. The processing device 250 is an example of the processing device 3. The microphone 241 is an example of the input device 4. The projection device 221 and the projection device 222 are examples of the display device 5. The storage device 270 is an example of the storage device 6.

[0121] In the example shown in the figure, the MR device 200 is a binocular head mounted display. Two lenses 211 and 212 are embedded in a frame 201. Projection devices 221 and 222 project information onto the lenses 211 and 212, respectively.

[0122] Projection devices 221 and 222 display the recognition result of the operator's hand, virtual objects, etc. on lenses 211 and 212. Only one of projection devices 221 and 222 may be provided, and information may be displayed on only one of lenses 211 and 212.

[0123] The lenses 211 and 212 are light-transmissive. The operator can visually recognize the actual situation through the lenses 211 and 212. In addition, the operator can also visually recognize the information projected by the projection devices 221 and 222 onto the lenses 211 and 212. The information is displayed in a superimposed manner in the actual space through the projection of the projection devices 221 and 222.

[0124] The image camera 231 detects visible light and obtains a two-dimensional image. The depth camera 232 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 240 is a 6-axis detection sensor capable of detecting 3-axis angular velocity and 3-axis acceleration. The microphone 241 receives voice input.

[0125] The processing device 250 controls each element of the MR device 200. For example, the processing device 250 controls the display of the projection device 221 and the projection device 222. The processing device 250 detects the movement of the field of view based on the detection result of the sensor 240. The processing device 250 changes the display of the projection device 221 and the projection device 222 according to the movement of the field of view. In addition, the processing device 250 can perform various processes using data obtained from the image camera 231 and the depth camera 232, data from the storage device 270, and the like.

[0126] The battery 260 supplies power required for the operation of each element of the MR device 200. The storage device 270 stores data required for processing by the processing device 250, data obtained by processing by the processing device 250, and the like. The storage device 270 may be provided outside the MR device 200 and communicate with the processing device 250.

[0127] The MR device according to the embodiment is not limited to the example shown in the figure, and may be a single-eye head mounted display. The MR device may be a glasses type as shown in the figure, or a helmet type.

[0128] Fig.16 It is a schematic diagram which illustrates the object of the work.

[0129] For example, for Fig.16 The article 300 shown is being fastened. The article 300 is a hollow cylindrical member having fastening positions 301 to 308. The worker uses a wrench to fasten screws at each of the fastening positions 301 to 308.

[0130] A marker 310 is provided near the work object. In the illustrated example, the marker 310 is an AR marker. As described later, the marker 310 is provided in order to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), etc. may be used as the marker 310. Alternatively, instead of a marker, a base point may be indicated by a gesture. The processing device 250 sets a three-dimensional coordinate system based on a plurality of points indicated by gestures.

[0131] Figures 17 to 20 It is a schematic diagram used to illustrate the processing performed by the mixed reality device involved in the implementation method.

[0132] When the fastening work starts, the image camera 231 and the depth camera 232 take a picture of the marker 310. The processing device 250 recognizes the marker 310 from the photographed image. The processing device 250 sets a three-dimensional coordinate system based on the position of the marker 310.

[0133] The image camera 231 and the depth camera 232 take photos of the object 300, the left hand of the operator, and the right hand of the operator. The processing device 250 recognizes the left hand and the right hand from the photographed images. The processing device 250 displays the recognition result to the lenses 211 and 212 via the projection devices 221 and 222.

[0134] For example Fig.17 As shown, the processing device 250 displays the recognition results of the left hand 351 and the recognition results of the right hand 352 on the hands in the real space in an overlapping manner. In the illustrated example, as the recognition results of the left hand 351 and the right hand 352, a plurality of virtual objects 361 and a plurality of virtual objects 362 are displayed. The plurality of virtual objects 361 respectively represent a plurality of joints of the left hand 351. The plurality of virtual objects 362 respectively represent a plurality of joints of the right hand 352. Instead of the joints, virtual objects representing the surface shapes of the left hand 351 and the surface shapes of the right hand 352 may be displayed.

[0135] Furthermore, the processing device 250 is as follows Fig.18 As shown, virtual objects 401 to 408 corresponding to the fastening positions 301 to 308 are displayed. The virtual objects 401 to 408 respectively indicate the positions where the operator's hands should be placed when fastening the fastening positions 301 to 308.

[0136] The positions where virtual objects 401 to 408 are displayed are pre-registered using a coordinate system based on the base point of the marker 310. For example, the virtual object is displayed at a position away from the coordinates of each fastening position by a predetermined distance. The "predetermined distance" is set in consideration of the length of the screw, the size of the wrench head, etc.

[0137] In the example shown in the figure, the virtual objects 401 to 408 are spheres. As long as the operator can visually recognize the virtual objects in a manner distinguishable from the real space, the shape, color, etc. of each virtual object may be arbitrary.

[0138] As an example, the operator tightens the screw at the tightening position 303. In this case, the operator places the screw in the threaded hole of the tightening position 303. Next, the operator holds the handle of the wrench 120 with the right hand. The operator inserts the screw into the front end (head) of the wrench 120 with the socket installed. Fig.19 As shown, the operator presses the head of the wrench 120 with the left hand 351 and rotates the wrench 120 with the right hand 352. Thus, the screw is tightened.

[0139] In the above series of actions, the processing device 250 determines whether the left hand 351 or the right hand 352 is in contact with a certain virtual object. In the above operation example, when tightening the screw, the left hand 351 pressing the head of the wrench is in contact with the virtual object 403. If the processing device 250 determines that the left hand 351 is in contact with the virtual object 403, it is inferred that the screw is being screwed at the tightening position 303 corresponding to the virtual object 403.

[0140] Alternatively, the contact between the hand and the virtual object can be replaced by judging whether the tool has contacted a virtual object. Fig.19 As shown, a plurality of markers 121 are mounted on the wrench 120. The processing device 250 recognizes the plurality of markers 121 from the image. The processing device 250 calculates the position of the head of the wrench 120 based on the positional relationship of the plurality of markers 121. The processing device 250 uses the calculated position of the head to determine whether the wrench 120 has come into contact with the virtual object.

[0141] Based on the distance between the specified object and the virtual object, the contact between the specified object and the virtual object is determined. The processing device 250 calculates the distance between the coordinates of the virtual object and the coordinates of the specified object. When the distance is less than a preset threshold, the processing device 250 determines that the specified object is in contact with the virtual object.

[0142] As an example, a threshold value such as Fig.18 and Fig.19As shown, it is represented by the size of the virtual object. In this example, the center of the sphere is located at the coordinates of the virtual object. The diameter of the virtual object is equivalent to the threshold. When a specified object in the real space overlaps with a virtual object in the virtual space, the distance between the virtual object and the specified object is less than the threshold. Therefore, it is determined that the specified object is in contact with the virtual object.

[0143] If the work position is estimated, the extraction of time series data, labeling of the extracted time series data, and judgment of the time series data are performed based on the estimation result. If the first judgment device 51 judges that the time series data is abnormal, for example Fig. 20 As shown, an alarm 450 is displayed near the wrench 120 or the hand. The operator can check the alarm 450 while performing the work.

[0144] By using the MR device, the operator can visually recognize information output from the processing system 10 while performing work. The operator does not need to move the face or field of vision to confirm the information output from the processing system 10. Therefore, the operator's work efficiency can be further improved.

[0145] Fig.21 It is a schematic diagram used to illustrate other processing performed by the mixed reality device involved in the implementation method.

[0146] In the above method, the tightening position of the screw is inferred based on the contact between the virtual object and the specified object. For example, the tightening position of the screw is inferred based on the contact between the hand pressing the head of the tool and the virtual object. Alternatively, the tightening position of the screw can be inferred based on the movement of the hand holding the handle of the tool.

[0147] For example Fig.21 As shown, while the wrench 120 is used to tighten the screw 125, the hand holding the handle moves in an arc shape. The center of the arc is located at the head of the wrench 120. The processing device 250 calculates the center coordinates of the rotation of the tool using the hand movement. When calculating the center coordinates, any of the following first to third methods can be used.

[0148] Figure 22 to Figure 24 This is a diagram for explaining the method of calculating the center coordinates.

[0149] In the first method, the processing device 250 extracts four different coordinates from the measured multiple coordinates. The processing device 250 calculates a sphere passing through the four coordinates. The sphere is represented by a spherical formula. Fig. 22As shown, the four coordinates are set to P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3) and P4 (x4, y4, z4). The coordinates of the center of the sphere are set to P0 (x0, y0, z0). The radius of the sphere is set to r. r is equivalent to the distance from the head of the wrench 120 to the hand (handle), which is pre-registered. In this case, the spherical formula is expressed by the following mathematical formula (1). In mathematical formula (1), k, l, m and n are constants.

[0150] [Mathematical formula 1]

[0151] (x-x0) 2 +(y-y0) 2 +(z-z0) 2 =r 2

[0152] Substituting the coordinate P1 into the equation (1) yields the following equation (2). The equation (2) is rewritten as the equation (3).

[0153] [Mathematical formula 2]

[0154] (x1-x0) 2 +(y1-y0) 2 +(z1-z0) 2 =r 2

[0155] [Mathematical formula 3]

[0156] x0 2 +y0 2 +z0 2 -2x1x0-2y1y0-2z1z0+x1 2 +y1 2 +z1 2 =0

[0157] Similarly, by substituting the coordinates P2 to P4 into the equation (1), the following equations (4) to (6) are obtained.

[0158] [Formula 4]

[0159] x0 2 +y0 2 +z0 2 -2x2x0-2y2y0-2z2z0+x2 2 +y2 2 +z2 2 =0

[0160] [Formula 5]

[0161] x0 2 +y02 +z0 2 -2x3x0-2y3y0-2z3z0+x3 2 +y3 2 +z3 2 =0

[0162] [Mathematical formula 6]

[0163] x0 2 +y0 2 +z0 2 -2x4x0-2y4y0-2z4z0+x4 2 +y4 2 +z4 2 =0

[0164] Solve the simultaneous equations of equations (3) to (6) to calculate the coordinates P0 (x0, y0, z0). Thus, the center of the sphere passing through the four coordinates P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3), and P4 (x4, y4, z4) is obtained. The processing device 250 uses the coordinates P0 (x0, y0, z0) of the center of the sphere as the center coordinates of the rotation of the wrench 120.

[0165] The processing device 250 may also extract a combination of multiple four coordinates. The processing device 250 calculates the coordinates of the center of the sphere for each combination. Thus, the coordinates of the centers of the multiple spheres are calculated. There is an error in the measured coordinates of the hand. Therefore, the coordinates of the multiple centers calculated are also different from each other. The processing device 250 uses the multiple coordinates to determine the coordinate used as the rotation center of the wrench 120. For example, the processing device 250 calculates the median of the multiple coordinates as the center coordinate of the rotation of the wrench 120. The processing device 250 may also calculate the average value or mode value of the multiple coordinates as the center coordinate of the rotation of the wrench 120. Thus, the accuracy of the calculated center coordinate can be improved.

[0166] In the second method, the processing device 250 extracts three coordinate pairs from the measured multiple coordinates. Each pair consists of two coordinates. Fig.23 In the example shown, a pair of coordinates P1 and P2, a pair of coordinates P3 and P4, and a pair of coordinates P5 and P6 are extracted. The six coordinates are respectively set to P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3), P4 (x4, y4, z4), P5 (x5, y5, z5), and P6 (x6, y6, z6). The processing device 250 calculates the midpoint of the two coordinates for each pair. The coordinates of the midpoint of each pair are respectively set to P 12 , P 34 and P 56 .

[0167] As described above, the hand moves in an arc shape. Therefore, the perpendicular bisector of each coordinate pair passing through each midpoint passes through the center of the circle. The center of the circle is set to coordinate P0 (x0, y0, z0). In this case, as shown in mathematical formula (7), the vector from coordinate P1 to coordinate P2 is the same as the vector from coordinate P 12 The inner product of the vector to the coordinate P0 is 0. The mathematical formula (7) can be rewritten as the mathematical formula (8).

[0168] [Formula 7]

[0169] (P2-P1)·(P0-P 12 )=0

[0170] [Mathematical formula 8]

[0171] (x2-x1)(x0-(x1+x2) / 2)+(y2-y1)(y0-(y1+y2) / 2)+(z2-z1)(z0-(z1+z2) / 2)=0

[0172] The following mathematical formulas (9) and (10) are obtained similarly for other coordinate pairs.

[0173] [Mathematical formula 9]

[0174] (x4-x3)(x0-(x3+x4) / 2)+(y4-y3)(y0-(y3+y4) / 2)+(z4-z3)(z0-(z3+z4) / 2)=0

[0175] [Formula 10]

[0176] (x6-x5)(x0-(x5+x6) / 2)+(y6-y5)(y0-(y5+y6) / 2)+(z6-z5)(z0-(z5+z6) / 2)=0

[0177] These simultaneous equations are solved to calculate the coordinates P0 (x0, y0, z0). The processing device 250 calculates the coordinates P0 (x0, y0, z0) as the center coordinates of the rotation of the wrench 120.

[0178] The processing device 250 may also extract more than four coordinate pairs. The processing device 250 selects three coordinate pairs from the four or more coordinate pairs, and calculates the coordinate P0 for each combination of the selected coordinate pairs. The processing device 250 calculates the median, average, or mode of the plurality of coordinates P0 as the center coordinate of the rotation of the wrench 120. Thus, the accuracy of the calculated center coordinate can be improved.

[0179] In the third method, the processing device 250 extracts three different coordinates from the measured multiple coordinates. The processing device 250 calculates the circumcenter O of the three coordinates. Fig.24 As shown, the three coordinates are set to P1 (x1, y1, z1), P2 (x2, y2, z2) and P3 (x3, y3, z3). The coordinates of the circumcenter O are set to P0 (x0, y0, z0). In addition, for the triangle obtained by connecting the coordinates P1 to P3, the length of the opposite side of the coordinate P1 is set to L1. The length of the opposite side of the coordinate P2 is set to L2. The length of the opposite side of the coordinate P3 is set to L3. The angle at the coordinate P1 is set to α. The angle at the coordinate P2 is set to β. The angle at the coordinate P3 is set to γ. In this case, the coordinates of the circumcenter O are expressed by the following mathematical formula (11). In addition, in mathematical formula (11), the symbol with an arrow attached indicates that it is a position vector. Mathematical formula (11) can be rewritten as mathematical formula (12). Mathematical formula (12) can be decomposed into mathematical formulas (13) to (15).

[0180] [Mathematical formula 11]

[0181]

[0182] [Mathematical formula 12]

[0183]

[0184] [Mathematical formula 13]

[0185]

[0186] [Formula 14]

[0187]

[0188] [Mathematical formula 15]

[0189]

[0190] Based on equations (13) to (15), x0, y0, and z0 are calculated respectively. The processing device 250 calculates the coordinates P0 (x0, y0, z0) of the circumcenter O as the center coordinates of the rotation of the wrench 120.

[0191] The processing device 250 may also extract a plurality of combinations of three coordinates. The processing device 250 calculates the coordinates of the circumcenter for each combination. The processing device 250 calculates the median, average, or mode of the plurality of coordinates as the center coordinate of the rotation of the wrench 120. Thus, the accuracy of the calculated center coordinate can be improved.

[0192] Before executing any one of the first to third methods, a portion of the coordinates of the hand may be selected. Based on the selected coordinates, one of the first to third methods may be executed. For example, only the coordinates inferred to be when the hand moves in an arc shape may be selected, and the coordinates used in the first to third methods may be extracted based on these coordinates. By using only the coordinates when the hand moves in an arc shape, the accuracy of the center coordinates can be further improved.

[0193] As an example, when a digital tool is used, the processing device 250 selects the coordinates of the hand obtained at the timing (moment) when data is received from the digital tool. If a screw is screwed using a digital tool, the digital tool detects torque, rotation angle, etc. The processing device 250 receives these detection values ​​detected by the digital tool. The reception of the detection values ​​shows that the hand moves in an arc shape. Therefore, by selecting the coordinates of the hand obtained at the timing (moment) when the detection values ​​are received and using them to calculate the center coordinates, the center coordinates can be calculated with higher accuracy.

[0194] Furthermore, even when the coordinates are not selected, the center coordinates can be calculated by calculating a plurality of candidates for the center coordinates in any of the first to third methods. For example, the processing device 250 excludes the deviation value from the plurality of candidates and calculates the median, mean, or mode value of the remaining candidates as the center coordinates.

[0195] The center coordinate of the rotation of the wrench 120 can be regarded as the position of the screw screwed by the wrench 120. By inferring the position of the screw, the tightening position of the screw can also be inferred. For example, in the case where the position of the tightening position is pre-registered, it is possible to tighten the screw at the tightening position closest to the center coordinate. A threshold is set for the distance between the position of the tightening position and the center coordinate. The processing device 250 calculates the distance between the center coordinate and the position of the tightening position closest to the center coordinate. When the distance is below the pre-set threshold and the detection value is received from the digital tool, the processing device 250 infers that the screw is being tightened at the nearest tightening position.

[0196] Once the work position is estimated, the processing device 250 extracts data when the screw is screwed into the work position from the time series data based on the time when the work was performed and the time when each torque was detected, thereby associating the extracted time series data with the data on the work position.

[0197] For example, in Fig.10After step S0 shown, the center coordinates are calculated using the coordinates of the hand obtained in the prescribed time length. When the screw is not tightened, the calculated center coordinates are away from the fastening position. As a result, it is inferred that the screw is not tightened. The processing device 250 offsets the time length and repeatedly calculates the center coordinates. If the screw is tightened, the center coordinates are close to the position of the fastening position. Furthermore, when a detection value is received from the digital tool, it is inferred that the screw is being tightened at the nearest fastening position. As an example, the time length is set to 6 seconds. The processing device 250 offsets the time length by 16 milliseconds each time and repeatedly calculates the center coordinates. The amount of offset is appropriately set according to the computing power of the processing device 250.

[0198] The processing device 250 may also infer the start time of the operation. For example, when the start of the operation has not yet been inferred, the processing device 250 infers the earliest time when the detection value is obtained by the digital tool within the specified time length as the start time of the operation. After inferring the start time of the operation, the processing device 250 determines the end time of the operation. When the distance between the center coordinate and the position of the nearest fastening position exceeds a threshold, the processing device 250 infers the earliest time when the detection value is no longer obtained from the digital tool within the time length as the end time of the operation.

[0199] The processing device 250 extracts data from the estimated start time to the estimated end time from the overall time series data. The processing device 3 associates the extracted time series data with data about the work location (e.g., ID). Thereafter, it is also possible to determine whether the extracted time series data is normal. The extracted time series data can also be used to determine the proficiency of the operator.

[0200] Figure 25 to Figure 27 This is a diagram for explaining the method of calculating the center coordinates.

[0201] The processing device 250 may also use the length of the tool between the wrench 120 and the screw to estimate the position of the screw more accurately. Figure 5 to Figure 7 In the example shown, a sleeve (not shown) is embedded in the wrench 120. That is, the center coordinate of the rotation of the wrench 120 is separated from the position of the screw by the length of the sleeve. When the length of the sleeve is pre-registered, the processing device 250 can use the center coordinate and the length of the sleeve to more accurately infer the position of the screw.

[0202] When the position of the screw is estimated using the length of the sleeve, it is necessary to determine on which side the screw is located relative to the rotation plane of the wrench 120. Fig.25 In the example shown in (a), the wrench 120 is rotated in the rotation direction RD1. The screw 125 and the sleeve 128 are located at the bottom. Fig.25In the example shown in (b), the wrench 120 rotates in the rotation direction RD2. The rotation direction RD2 is opposite to the rotation direction RD1. The screw 125 and the sleeve 128 are located on the upper side with respect to the plane parallel to the rotation direction RD2.

[0203] In order to determine which side the screw 125 is located, the processing device 250 uses the center coordinates, the two coordinates of the hand, the time series information of the two coordinates, and the tightness information of the screw. Fig.26 As shown, the two coordinates are set to P1 (x1, y1, z1) and P2 (x2, y2, z2). The center coordinate is set to P0 (x0, y0, z0). The time when the hand is at the coordinate P1 and the time when the hand is at the coordinate P2 are known. That is, the processing device 250 maintains the time series information of the coordinates P1 and P2. In this example, the time when the hand is at the coordinate P1 is earlier than the time when the hand is at the coordinate P2.

[0204] The tightness information indicates whether the screw is being tightened or loosened. In the case where the wrench 120 is a digital tool, the wrench 120 determines whether the screw is being tightened or loosened based on the detected torque and generates the tightness information. The processing device 250 can also determine whether the screw is being tightened or loosened based on the time series data of the torque received from the wrench 120 and generate the tightness information.

[0205] The plane passing through the coordinates P0 to P2 is expressed by the following mathematical formula (16). In mathematical formula (16), k, l, m, and n are constants.

[0206] [Formula 16]

[0207] kx+ly+mz+n=0

[0208] Substituting the coordinates of P0 to P2 into equation (16), the following equations (17) to (19) are obtained. Based on equations (17) to (19), constants k, l, m, and n are calculated.

[0209] [Mathematical formula 17]

[0210] kx0+ly0+mz0+n=0

[0211] [Mathematical formula 18]

[0212] kx1+ly1+mz1+n=0

[0213] [Mathematical formula 19]

[0214] kx2+ly2+mz2+n=0

[0215] Here, the processing device 250 calculates a vector from the center coordinate P0 to the coordinate P1 at the previous time. In addition, the processing device 250 calculates a vector from the center coordinate P0 to the coordinate P2 at the next time. The screw 125 is located at a position P0 on the normal vector P0P1×P0P2, which is a distance L0 from the center coordinate P0. Q When the screw is tightened and the time of the coordinate P1 is earlier than the time of the coordinate P2, the processing device 250 calculates the normal vector P0P1×P0P2 between the vector P0P1 and the vector P0P2.

[0216] From coordinate P0 to coordinate P where the wrench and socket act on the screw Q The length to is expressed by the following mathematical formula (20). In the following mathematical formulas, an arrow added to a symbol indicates that the value represented by the symbol is a vector.

[0217] [Mathematical formula 20]

[0218]

[0219] On the other hand, from coordinate P0 to coordinate P Q The vector of is also expressed by the following mathematical formula (21) using vector P0P1 and vector P0P2. In mathematical formula (21), t is a constant.

[0220] [Mathematical formula 21]

[0221]

[0222] Substituting equation (21) into equation (20) yields equation (22) below. Length L0 in equation (22) is pre-registered. By solving equation (22), t is calculated.

[0223] [Mathematical formula 22]

[0224] √{({(y1-y0)(z2-z0)-(y2-y0)(z2-z0)} 2 *t 2 +{(z1-z0)(x2-x0)-(z2-z0)(x1-x0)}*t 2 +{(x1-x0)(y2-y0)-(x2-x0)(y1-y0)}*t 2 =L0

[0225] If t is calculated, use the coordinate P0, constants k, l, m, n and t to calculate the coordinate P Q . That is, the position of the screw is obtained.

[0226] like Fig. 27As shown, the screw may be tightened via an extension rod 130. The extension rod 130 is installed between the wrench 120 and the screw. In this case, the position of the screw can also be estimated using the length of the extension rod 130 in the same manner as the above-mentioned method.

[0227] That is, the screw is located on the normal vector P0P1×P0P2 at a position that is the sum of the length of the sleeve 128 and the length of the extension rod 130 away from the center coordinate P0. The position P of the screw is estimated using the center coordinate P0, the length of the sleeve 128, and the length of the extension rod 130. Q By taking into account the length of other tools between the screw and the wrench 120, the position of the screw can be inferred with greater accuracy.

[0228] When the position of the screw is estimated, the processing device 250 estimates the tightening position where the screw is screwed using the estimated position of the screw. For example, the processing device 250 calculates the distance between the position of the screw and the position of the tightening position closest to the screw. When the distance is less than a preset threshold, the processing device 250 estimates that the screw is being tightened at the nearest tightening position.

[0229] Fig.28 It is a schematic diagram illustrating a working situation.

[0230] For example Fig.28 As shown in the figure, the fastening position 141 is sometimes present in a small place. It is difficult for the operator to visually identify the fastening position 141. Furthermore, in this example, the left hand cannot be inserted into the place of the fastening position, and the operator cannot hold the head of the wrench 120 with the left hand. In other words, the hand cannot be brought into contact with the object. In addition, even if the hand is in contact with the object, the image camera 231 and the depth camera 232 cannot photograph the situation. Therefore, the contact between the hand and the object cannot be detected. In such a case, by calculating the center coordinates of the rotation of the tool based on the movement of the hand, the position where the screw is fastened can be inferred.

[0231] In addition to calculating the center coordinates, it is also possible to detect the contact of the object with a predetermined object. Compared with calculating the center coordinates, the amount of calculation required for contact with the object is less. Therefore, by detecting the contact of the object with the object, the position where the screw is tightened can be inferred more quickly. For example, in the case of a work error, by outputting an alarm earlier, the situation of re-doing the work can be reduced.

[0232] In the examples described above, the example of tightening a screw at a fastening position is mainly described. The embodiments of the present invention are applicable not only to the case of tightening a screw at a fastening position, but also to the case of loosening a screw at a fastening position. For example, when performing maintenance, inspection or repair of a product, the screw at the fastening position is loosened. According to the embodiments of the present invention, when loosening the screw, the extracted time series data can be associated with the data on the working position.

[0233] Fig.29 This is a schematic diagram showing the hardware configuration.

[0234] As the processing device 3, the learning device 8 and the processing device 250, for example, Fig.29 The computer 90 shown in FIG. 9A includes a CPU 91 , a ROM 92 , a RAM 93 , a storage device 94 , an input interface 95 , an output interface 96 , and a communication interface 97 .

[0235] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for the computer 90 to implement the above-mentioned processing. The RAM 93 functions as a storage area for developing the program stored in the ROM 92.

[0236] The CPU 91 includes a processing circuit. The CPU 91 uses the RAM 93 as a working memory and executes a program stored in at least one of the ROM 92 and the storage device 94. While executing the program, the CPU 91 controls each component via the system bus 98 and executes various processes.

[0237] The storage device 94 stores data required for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) etc. The storage device 94 can also be used as the storage device 6 or the storage device 270.

[0238] An input interface (I / F) 95 can connect the computer 90 to the input device 4. The CPU 91 can read various data from the input device 4 via the input I / F 95.

[0239] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can send data to the display device 5 via the output I / F 96 and cause the display device 5 to display information.

[0240] The communication interface (I / F) 97 can connect a device outside the computer 90 to the computer 90. The communication I / F 97 connects a digital tool to the computer 90 through, for example, Bluetooth (registered trademark) communication.

[0241] The data processing performed by the processing device 3 or the processing device 250 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0242] The above-mentioned various data processing can also be recorded as a program that can be executed by a computer, on a disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or other non-volatile computer-readable recording media (non-transitory computer-readable storage medium).

[0243] For example, the information recorded in the recording medium can be read out by a computer (or a built-in system). In the recording medium, the recording format (storage format) is arbitrary. For example, the computer reads out a program from the recording medium, and based on the program, the instructions recorded in the program are executed by the CPU. In the computer, the program can also be obtained (or read out) via a network.

[0244] (Feature 1)

[0245] A processing device, wherein

[0246] Using the first image, the first work position where the screw is tightened is estimated.

[0247] extracting first time series data when the screw is screwed at the first working position from time series data acquired by a tool for screwing the screw,

[0248] The first time-series data is associated with data on the first work position.

[0249] (Feature 2)

[0250] The processing device according to feature 1, wherein

[0251] Based on the first image, the coordinates of the hand of the operator who is tightening the screw are measured;

[0252] The first work position is estimated using the coordinates.

[0253] (Feature 3)

[0254] The processing device according to feature 2, wherein:

[0255] The data of the operator and the time when the screw was tightened are also associated with the data on the first working position.

[0256] (Feature 4)

[0257] The processing device according to any one of features 1 to 3, wherein

[0258] Determine whether the first time series data is normal,

[0259] When the first time-series data is determined to be abnormal, an alarm is output.

[0260] (Feature 5)

[0261] A processing device as described in feature 4, wherein:

[0262] A determination result of whether the first time-series data is normal is also associated with the data on the first work position.

[0263] (Feature 6)

[0264] A processing device as described in feature 4 or 5, wherein:

[0265] The first time series data includes torque,

[0266] When the torque exceeds the first threshold value, or when the period of the first time-series data is longer than the second threshold value, the first time-series data is determined to be abnormal.

[0267] (Feature 7)

[0268] A processing device as described in feature 4 or 5, wherein:

[0269] inputting the first time series data into the first judgment device,

[0270] Based on the result output from the first determiner, it is determined whether the first time-series data is normal.

[0271] (Feature 8)

[0272] The processing device according to any one of features 1 to 7, wherein

[0273] Using the second image, the second work position where another screw is tightened is estimated.

[0274] extracting, from the time series data, second time series data when the other screw is screwed at the second working position,

[0275] The second time-series data is associated with the data on the second work position.

[0276] (Feature 9)

[0277] The processing device according to any one of features 1 to 8, wherein

[0278] inputting the first time series data into a second determination device for determining the proficiency of a worker who screws the screws,

[0279] The proficiency of the operator is determined based on the result output from the second determination device.

[0280] (Feature 10)

[0281] A learning device, wherein

[0282] Using the data on the first work position and the first time-series data associated with each other by the processing device according to feature 1, a first determiner for determining whether the first time-series data is normal is trained.

[0283] (Feature 11)

[0284] A learning device, wherein

[0285] The second determiner for determining the proficiency of the operator who screws the screw is trained using the data on the first work position and the first time-series data associated with each other by the processing device according to feature 1.

[0286] (Feature 12)

[0287] A mixed reality device that displays a virtual space overlaid on an actual space, wherein:

[0288] Displaying a virtual object corresponding to the tightening position of the screw,

[0289] When the virtual object comes into contact with a predetermined object, at least a portion of the time series data acquired by the tool for tightening the screw is associated with the data on the fastening position corresponding to the virtual object that came into contact with the screw.

[0290] (Feature 13)

[0291] A processing method, wherein a computer is caused to execute:

[0292] Using the first image, the first work position where the screw is tightened is estimated.

[0293] extracting first time series data when the screw is screwed at the first working position from time series data acquired by a tool for screwing the screw,

[0294] The first time-series data is associated with data on the first work position.

[0295] (Feature 14)

[0296] A program for causing a computer to execute the processing method according to feature 13.

[0297] (Feature 15)

[0298] A storage medium storing the program according to feature 14.

[0299] In the present specification, "or" means that "at least one or more" of the items listed in the sentence can be adopted.

[0300] According to the above-described embodiments, a processing device, a processing system, a mixed reality device, a processing method, a program, and a storage medium are provided that can automatically associate data about a work position with corresponding time series data. For example, it is possible to determine which fastening position each part of the time series data detected by a tool is data for when a screw is screwed. In addition, according to the embodiments, a learning device is provided that can use the associated data to enable a determiner to learn.

[0301] Several embodiments of the present invention are illustrated above, but these embodiments are provided as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other ways, and various omissions, substitutions, changes, etc. can be made without departing from the scope of the subject matter of the invention. These embodiments and their modifications are included in the scope and subject matter of the invention, and are included in the invention described in the claims and their equivalents. In addition, the above-mentioned embodiments can be implemented in combination with each other.

Claims

1. A processing device, wherein: Using the first image, the first work position where the screw is tightened is estimated. extracting first time series data when the screw is screwed at the first working position from time series data acquired by a tool for screwing the screw, The first time-series data is associated with data on the first work position.

2. The processing device according to claim 1, wherein: Based on the first image, the coordinates of the hand of the operator who is tightening the screw are measured; The first work position is estimated using the coordinates.

3. The processing device according to claim 2, wherein: The data of the operator and the time when the screw was tightened are also associated with the data on the first working position.

4. The processing device according to claim 1, wherein: Determine whether the first time series data is normal, When the first time-series data is determined to be abnormal, an alarm is output.

5. The processing device according to claim 4, wherein: A determination result of whether the first time-series data is normal is also associated with the data on the first work position.

6. The processing device according to claim 4, wherein: The first time series data includes torque, When the torque exceeds the first threshold value, or when the period of the first time-series data is longer than the second threshold value, the first time-series data is determined to be abnormal.

7. The processing device according to claim 4, wherein: inputting the first time series data into the first judgment device, Based on the result output from the first determiner, it is determined whether the first time-series data is normal.

8. The processing device according to any one of claims 1 to 7, wherein: Using the second image, the second work position where another screw is tightened is estimated. extracting, from the time series data, second time series data when the other screw is screwed at the second working position, The second time-series data is associated with data on the second work position.

9. The processing device according to any one of claims 1 to 7, wherein: inputting the first time series data into a second determination device for determining the proficiency of a worker who screws the screws, The proficiency of the operator is determined based on the result output from the second determination device.

10. A learning device, wherein: Using the data on the first work position and the first time-series data associated with each other by the processing device according to claim 1, a first determiner for determining whether the first time-series data is normal is trained.

11. A learning device, wherein: The second determiner for determining the proficiency of the operator who screws the screw is trained using the data on the first work position and the first time-series data associated with each other by the processing device according to claim 1 .

12. A mixed reality device that displays a virtual space overlaid on an actual space, wherein: Displaying a virtual object corresponding to the tightening position of the screw, When the virtual object comes into contact with a predetermined object, at least a portion of the time series data acquired by the tool for tightening the screw is associated with the data on the fastening position corresponding to the virtual object that came into contact with the screw.

13. A processing method, wherein: Causes the computer to execute: Using the first image, the first work position where the screw is tightened is estimated. extracting first time series data when the screw is screwed at the first working position from time series data acquired by a tool for screwing the screw, The first time-series data is associated with data on the first work position.

14. A program for causing a computer to execute the processing method according to claim 13.

15. A storage medium storing the program according to claim 14.

Images