Data generation device, work determination system, data generation method and program

Abstract

The processing device (40) is a device that generates data for making judgments regarding the work to be performed from an image of the work to be performed obtained by photographing the work to be performed by a worker, and includes an image acquisition unit (42) that acquires multiple standard images obtained by photographing the usage status of the tool in standard work performed as a standard for the work to be performed using an imaging unit (11) attached to the tool, a distance acquisition unit (45) that acquires distance information indicating the distance between the tool shown in each standard image acquired by the image acquisition unit (42) and the work in the standard work in which the tool is used, and a generation unit (47) that extracts two or more standard images from the multiple standard images, the distances of which are different from each other and which are acquired by the distance acquisition unit (45), as extracted images, and generates data from the extracted images.

Description

[Technical Field]

[0001] The present disclosure relates to a data generation device, an operation determination system, a data generation method, and a program. [Background technology]

[0002] In a factory, the same tasks are repeatedly performed on the same objects. In order to perform these tasks appropriately, it is desirable to manage the tasks to be performed and make them uniform (see, for example, Patent Document 1).

[0003] Patent Document 1 describes a technology that calculates the position of a tool by photographing the work area where the tool is to be performed using a marker-attached tool and determines whether the position is appropriate. This technology makes it possible to manage the position of the tool during work performed by a worker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-132736 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, the entire work area, including the tool, is photographed by a fixed camera. However, when the tool is inserted into a narrow, complicated space, the marker is positioned in the blind spot of the camera, making it impossible to calculate the position of the marker. Therefore, it is necessary to consider the mounting position of the camera and the installation position of the marker so that the marker does not enter the blind spot of the camera, but there may be cases where such a position does not exist in the first place.

[0006] When a camera is attached to a tool, the position of the tool at the time of photographing can be identified from the photographed image. Specifically, by preparing in advance a number of images that can be photographed in the work area, and comparing the images with the prepared images, or by learning a model that identifies the photographed position from the prepared images, it becomes possible to identify the position from the photographed image.

[0007] However, if the multiple images prepared in advance are inappropriate, the accuracy of the identified position will be low, so it is necessary to prepare appropriate images. However, manually preparing such images one by one can be a cumbersome preparation work. Therefore, there is room to reduce the burden of preparation work for managing the work by taking photos.

[0008] The present disclosure has been made in light of the above-mentioned circumstances, and aims to reduce the burden of preparatory work for managing work by taking photographs. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the data generation device disclosed herein is a data generation device that generates data for making judgments regarding the work to be performed based on an image of the work to be performed that is obtained by photographing the work to be performed, and is equipped with: an image acquisition means that acquires multiple standard images obtained by photographing the usage of the tool in standard work that is performed as a standard for the work to be performed using an imaging means attached to the tool; a distance acquisition means that acquires distance information indicating the distance between the tool shown in each standard image acquired by the image acquisition means and the work object in the standard work in which the tool is used; and a generation means that extracts two or more standard images from the multiple standard images that have different distances acquired by the distance acquisition means as extracted images, and generates data from the extracted images. [Effects of the Invention]

[0010] According to the present disclosure, the burden of preparation work for managing work can be reduced by taking photographs. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a configuration of an operation determination system according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of displaying a result of a determination regarding a task according to an embodiment. [Figure 3] FIG. 1 is a diagram showing a hardware configuration of a calculation device according to an embodiment. [Figure 4] FIG. 1 is a diagram showing a functional configuration of an operation determination system according to an embodiment. [Figure 5] FIG. 1 is a diagram for explaining a trigger according to an embodiment. [Figure 6] FIG. 1 is a diagram for explaining clarity according to an embodiment. [Figure 7] FIG. 1 is a diagram for explaining distance estimation according to an embodiment; [Figure 8] FIG. 10 is a diagram showing an example of a change in distance according to an embodiment; [Figure 9] FIG. 10 is a diagram showing an example of information stored in a buffer according to an embodiment; [Figure 10] FIG. 10 is a diagram for explaining the addition of data from a buffer to training data according to an embodiment. [Figure 11] FIG. 10 is a diagram showing an example of learning data according to an embodiment; [Figure 12] FIG. 1 is a diagram for explaining a classification model according to an embodiment. [Figure 13] FIG. 1 is a diagram for explaining a tip determination model according to an embodiment. [Figure 14] FIG. 10 is a diagram for explaining estimation of the attitude of a tool according to an embodiment; [Figure 15] FIG. 1 is a diagram for explaining a physical state identification model according to an embodiment. [Figure 16] 1 is a flowchart showing a preparation process according to an embodiment; [Figure 17] 1 is a flowchart showing a preparation trigger process according to an embodiment; [Figure 18] 1 is a flowchart showing a learning process according to an embodiment; [Figure 19] 1 is a flowchart showing operation processing according to an embodiment; [Figure 20] 1 is a flowchart showing operation trigger processing according to an embodiment; [Figure 21] 1 is a flowchart showing a determination process according to an embodiment; [Figure 22] 1 is a flowchart showing a visual inspection process according to an embodiment; [Figure 23] FIG. 10 is a diagram showing the configuration of a work determination system according to a modified example. [Figure 24] FIG. 10 is a diagram showing an example of obtaining distance information according to a modified example; DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an operation determination system according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0013] Embodiment The work determination system 1000 according to this embodiment is a system that captures work performed by a worker using a tool in a factory with a camera attached to the tool and notifies the worker of the results of a determination regarding the work based on the captured image. The following description focuses on an example in which a screw is fastened to a workpiece using an electric screwdriver as the tool. However, the work using a tool is not limited to this, and may include applying a chemical agent using a dispenser, welding using a welding torch, or other work.

[0014] As shown in FIG. 1, the work judgment system 1000 includes a tool 10 used by workers 81 and 82, a processing device 40 that generates learning data from images captured in a preparation phase, a learning device 50 that learns judgment data 500 from the learning data, which includes a judgment model for making judgments about the work, a UI (User Interface) terminal 60 that notifies the worker 82 of the results of judgment made by the processing device 40 based on the judgment data 500 in an operation phase, and a higher-level device 70 that collects the results of the judgment.

[0015] The tool 10 is connected to the processing device 40 via a wire 14 (described later) so as to be able to communicate with the processing device 40. The processing device 40, the learning device 50, the UI terminal 60, and the higher-level device 70 are connected to be able to communicate with each other via an information network such as an industrial network or a local area network (LAN), but may also be connected via a communication line. All or part of the communication between the tool 10, the processing device 40, the learning device 50, the UI terminal 60, and the higher-level device 70 may be wireless communication.

[0016] The processing device 40 may be a PLC (Programmable Logic Controller), the learning device 50 may be an industrial PC (Personal Computer), and the UI terminal 60 may be a programmable display, but is not limited to this. For example, the processing device 40 may be a single-board computer, and the learning device 50 may be a computing device on the Internet.

[0017] In the preparation phase, standard work equivalent to the work to be performed in the operation phase described below is performed, and images are taken during this standard work. In the standard work, a worker 81 uses a tool 10 to fasten screws 22a, 22b, 22c, and 22d in this order into four screw holes in the workpiece 21, thereby fixing the workpiece 21 to the workpiece 20.

[0018] The tool 10 includes an imaging unit 11 attached so that the imaging range includes the tip 12 and the object to be fastened, as shown by the dashed line, the tip 12 which is a removable bit, a switch 13 for rotating the tip 12, and a wire 14 for receiving power from an external source and communicating with the external device. The imaging unit 11 repeatedly captures images while receiving power and transmits the captured images to a processing device 40 via the wire 14. The wire 14 also transmits a signal indicating the switch state to the processing device 40 when the switch is switched ON or OFF. The tool 10 is used in combination with one tip selected from multiple types of tip parts that act on a workpiece.

[0019] In the operation phase, the worker 82 uses the tool 10 to fasten the screws 32a, 32b, 32c, and 32d in this order into the four screw holes in the workpiece 31, thereby fixing the workpiece 31 to the workpiece 30. While this work is being performed, the processing device 40 performs a determination regarding the work on the images captured by the photographing unit 11. The workpieces 20, 21, 30, and 31 correspond to examples of work objects.

[0020] 1, the same tool 10 is used in the preparation phase and the operation phase, but different tools 10 of the same type may be used in these phases. Also, in the operation phase, multiple tools 10 may be used simultaneously, and the processing device 40 may perform determination for each operation in which each tool 10 is used.

[0021] The results of the determination by the processing device 40 are displayed on the screen of the UI terminal 60, as shown in FIG. 2. In detail, on the left side of the screen, images of the works 30 and 31 are displayed, along with dashed frames representing the portion where an incorrect procedure was performed and the portion indicating the next procedure to be performed. On the right side of the screen, images taken for each procedure to be performed are displayed, and if the procedure was performed correctly, a check mark is added to the image, and if the procedure was performed incorrectly, an error message is displayed along with the image. Note that no image is displayed for "Procedure 4" in FIG. 2, as it has not yet been performed.

[0022] The processing device 40 and the learning device 50 have a hardware configuration as shown in Fig. 3. The arithmetic device 90 in Fig. 3 corresponds to each of the processing device 40 and the learning device 50. The arithmetic device 90 is configured as a computer having a processor 101, a main memory unit 102, an auxiliary memory unit 103, an input unit 104, an output unit 105, and a communication unit 106. The main memory unit 102, the auxiliary memory unit 103, the input unit 104, the output unit 105, and the communication unit 106 are all connected to the processor 101 via an internal bus 107.

[0023] The processor 101 includes a CPU (Central Processing Unit) as a processing circuit. The processor 101 executes a program P1 stored in the auxiliary storage unit 103 to realize various functions and execute the processes described below.

[0024] The main memory unit 102 includes a RAM (Random Access Memory). A program P1 is loaded into the main memory unit 102 from the auxiliary memory unit 103. The main memory unit 102 is used as a working area for the processor 101.

[0025] The auxiliary storage unit 103 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) and an HDD (Hard Disk Drive). In addition to the program P1, the auxiliary storage unit 103 stores various data used in the processing of the processor 101. The auxiliary storage unit 103 supplies the processor 101 with data used by the processor 101 in accordance with instructions from the processor 101. The auxiliary storage unit 103 also stores data supplied from the processor 101.

[0026] The input unit 104 includes input devices such as hardware switches, input keys, a keyboard, and a pointing device. The input unit 104 acquires information input by a user of the arithmetic device 90 and notifies the processor 101 of the acquired information.

[0027] The output unit 105 includes output devices such as a light emitting diode (LED), a liquid crystal display (LCD), and a speaker. The output unit 105 presents various information to the user in accordance with instructions from the processor 101.

[0028] The communication unit 106 includes a communication interface circuit for communicating with an external device. The communication unit 106 receives a signal from the outside and outputs data indicated by this signal to the processor 101. The communication unit 106 also transmits a signal indicating the data output from the processor 101 to the external device.

[0029] The tool 10 also has a configuration as a small-scale computer including a CPU and RAM. The photographing unit 11 of the tool 10 includes an image sensor and a lens, and is attached to the tool 10 after adjusting the attachment position, attitude, and viewing angle of the photographing unit 11 so that the part of the tip 12 that acts on the workpiece is captured in the center of the image. Note that a photographing unit having a specific viewing angle may be attached to the tool 10 at a predetermined attachment position and attitude without adjusting the attachment position, attitude, and viewing angle of the photographing unit 11.

[0030] The above-described hardware configurations work together to enable the tool 10, the processing device 40, and the learning device 50 to perform various functions. In detail, as shown in Fig. 4, the tool 10 has an image capturing unit 11, a trigger monitoring unit 15 that monitors whether or not a trigger corresponding to each procedure has occurred, and a communication unit 16 that transmits the captured image and a trigger signal.

[0031] The photographing unit 11 constantly photographs images of 640 x 480 pixels and sequentially transmits the photographed images to the processing device 40 via the communication unit 16. The trigger monitoring unit 15 is realized by a CPU and RAM. The trigger monitoring unit 15 monitors whether the switch 13 of the tool 10 has been switched, and when the switch 13 has been switched to an ON state or an OFF state, generates a trigger signal indicating this switching as a trigger and transmits the trigger signal to the processing device 40 via the communication unit 16. The communication unit 16 is realized by a network interface circuit. Note that the size of the image photographed by the photographing unit 11 is not limited to the above example and may be changed, for example, to correspond to the size of the target object.

[0032] As shown in FIG. 4, the processing device 40 also includes a trigger receiving unit 41 that receives a trigger signal from the tool 10, an image acquisition unit 42 that acquires an image from the tool 10, a clarity calculation unit 43 that calculates the clarity of the image, a distance estimation unit 44 that estimates the distance to the workpiece shown in the image, a distance acquisition unit 45 that acquires distance information indicating the distance, a buffer 46 in which images are stored, a generation unit 47 that generates learning data 501 in the preparation phase, a work judgment unit 48 that performs judgment regarding the work in the operation phase, a reception unit 49 that receives setting information 502, a display control unit 410 that controls the information displayed on the UI terminal 60, and a transmission unit 411 that transmits information indicating the result of the judgment to the higher-level device 70.

[0033] The trigger receiving unit 41 is mainly realized by the communication unit 106. As shown in FIG. 5 , the trigger receiving unit 41 receives from the tool 10 a trigger signal indicating an ON trigger, which indicates that the switch 13 has been switched to an ON state to tighten a screw in each step, and a trigger signal indicating an OFF trigger, which indicates that the switch 13 has been switched to an OFF state. In this manner, the trigger signals are generated at timings corresponding to each step. The trigger receiving unit 41 then generates a generation trigger, which is delayed from the OFF trigger by a predetermined length, to indicate the end of each step. The trigger receiving unit 41 outputs the trigger signal indicating the generation trigger to the generation unit 47 in the preparation phase and to the work determination unit 48 in the operation phase. Note that the trigger receiving unit 41 may output a trigger signal indicating an OFF trigger as an end of each step without generating a generation trigger. The trigger receiving unit 41 corresponds to an example of a trigger receiving means for receiving a trigger signal. The trigger receiving unit 41 also corresponds to an example of a trigger receiving means for receiving a trigger signal generated at a timing corresponding to each step.

[0034] The image acquisition unit 42 is mainly realized by the communication unit 106. The image acquisition unit 42 receives images transmitted from the tool 10 and outputs them to the clarity calculation unit 43. Because images are constantly captured by the tool 10, the image acquisition unit 42 sequentially acquires the captured images as indicated by the arrows tangent to the time axis in FIG. 6. The image capture and transmission may be continuous capture and sequential transmission of still images, or streaming transmission of moving images. The image acquisition unit 42 acquires both standard images captured in the preparation phase and actual images captured in the operation phase. The image acquisition unit 42 also corresponds to an example of an image acquisition means that repeatedly captures, with a photographing device attached to the tool, multiple actual images of the tool usage status in a task including multiple procedures performed using the tool. The image acquisition unit 42 also corresponds to an example of an image acquisition means that captures, with a photographing device attached to the tool, multiple standard images of the tool usage status in a standard task performed as a standard for the task.

[0035] Returning to FIG. 4 , the sharpness calculation unit 43 is primarily implemented by the processor 101. The sharpness calculation unit 43 calculates sharpness, which indicates the degree to which a subject is clearly captured in an image. Image "IMG01" shown schematically in FIG. 6 has a high sharpness, while image "IMG02" has a low sharpness because the subject, a tool and a workpiece, were photographed with blur. The sharpness may be an index value that quantifies the sharpness of an image, or may be another index value. The index value that quantifies the sharpness is calculated, for example, by normalizing the sum of pixel values ​​of contours detected by applying edge filtering to the image. If the calculated sharpness is higher than a predetermined threshold, the sharpness calculation unit 43 outputs the image having that sharpness to the distance estimation unit 44, thereby selecting the image for storage in the buffer 46. On the other hand, if the sharpness is lower than the threshold, the sharpness calculation unit 43 discards the image having that sharpness and excludes it from being stored in the buffer 46. The sharpness calculation unit 43 corresponds to an example of a sharpness calculation means that calculates sharpness indicating the degree to which the subject is clearly depicted in the actual image acquired by the image acquisition means, and stores the actual image having a sharpness exceeding a predetermined threshold in a buffer.

[0036] The distance estimation unit 44 is mainly realized by the processor 101. The distance estimation unit 44 recognizes all or part of the workpiece shown in the image and estimates the distance between the tool 10 and the workpiece. For example, as shown in FIG. 7, the distance estimation unit 44 recognizes a screw hole shown in the image and estimates the distance based on the size of the recognized screw hole on the image. A model or algorithm for estimating this distance is built into the distance estimation unit 44 in advance.

[0037] The distance between the tool 10 and the workpiece changes during the screw tightening procedure, as shown as distance D in FIG. 8. It is also believed that the change in this distance varies depending on the worker. Therefore, it is desirable that the learning data includes a variety of distances when the images are captured. The distance estimation unit 44 attaches distance information indicating the estimated distance to the image and outputs the image to the distance acquisition unit 45. The distance estimation unit 44 corresponds to an example of a distance estimation means that estimates the distance from the actual image acquired by the image acquisition means. The distance estimation unit 44 also corresponds to an example of a distance estimation means that estimates the distance from the standard image.

[0038] The distance acquisition unit 45 is primarily implemented by the processor 101. The distance acquisition unit 45 stores images whose distances indicated by distance information are within a predetermined range in the buffer 46, and discards images whose distances are outside this range. The predetermined range is, for example, 30 cm or less, which is a practical range for performing each task. The distance acquisition unit 45 corresponds to an example of a distance acquisition means that acquires distance information indicating the distance between a tool and a work object shown in each of the execution images acquired by the image acquisition means and stores execution images whose distances are within the predetermined range in the buffer. It also corresponds to an example of a distance acquisition means that acquires distance information indicating the distance estimated by the distance estimation means. The distance acquisition unit 45 also corresponds to an example of a storage means that stores execution images acquired by the image acquisition means in the buffer. The distance acquisition unit 45 also corresponds to an example of a distance acquisition means that acquires distance information indicating the distance between a tool shown in each of the standard images acquired by the image acquisition means and a work object in a standard task in which the tool is used.

[0039] The buffer 46 is realized by at least one of the main memory unit 102 and the auxiliary memory unit 103. As illustrated in Fig. 9, the buffer 46 stores a plurality of records that associate an image with the shooting time indicated by the image metadata, the image sharpness, and the distance estimated from the image. The buffer 46 corresponds to an example of a buffer that stores the actual image acquired by the image acquisition means.

[0040] The generation unit 47 is mainly realized by the processor 101. Upon receiving a trigger signal, the generation unit 47 reads data stored in the buffer 46, deletes the data in the buffer 46, and adds the read data to the training data 501. As a result, as shown in FIG. 10 , each time a procedure constituting a task is executed, the data read from the buffer 46 is added to the training data 501, and new data from the next procedure is accumulated in the buffer 46. Therefore, the capacity of the buffer 46 only needs to be large enough to store the data generated when one procedure is executed. Furthermore, the conditions for the images that the generation unit 47 should add from the buffer 46 to the training data 501 may be predetermined. For example, the generation unit 47 may select three types of records from the buffer 46, namely, those with the highest, lowest, and intermediate clarity, and add them to the training data 501.

[0041] The work determination unit 48 is primarily implemented by the processor 101. Upon receiving a trigger signal, the work determination unit 48 reads an image from the buffer 46 and applies the model learned by the learning device 50 to the image to determine the work status. For example, by applying the model, the work determination unit 48 identifies that the captured image contains the workpieces 30 and 31 corresponding to step c. Furthermore, the work determination unit 48 compares the determination result obtained by applying the model with the setting information 502 received from the receiving unit 49, as necessary. For example, if the workpieces 30 and 31 corresponding to step c are captured after step a, and the setting information 502 indicates that step b should be performed after step a, the work determination unit 48 determines that an incorrect work was performed in violation of the setting information 502. The work determination unit 48 then outputs the correctness of the work to the display control unit 410 and the transmission unit 411. The work determination unit 48 corresponds to an example of a determination means that, when a trigger signal is received by the trigger receiving means, makes a determination about a work based on the performed image stored in the buffer before another trigger signal corresponding to the next step of the procedure corresponding to the one trigger signal is received, and the processing device 40 having the work determination unit 48 corresponds to an example of a work determination device. Furthermore, the work determination unit 48 corresponds to an example of a determination means that makes a determination about the performed work using a model.

[0042] The reception unit 49 is mainly realized by the communication unit 106. The reception unit 49 receives setting information 502 from the learning device 50. However, the reception unit 49 may be realized by the input unit 104 and acquire setting information 502 input directly by the user of the processing device 40 instead of from the learning device 50. The reception unit 49 corresponds to an example of a reception means that receives setting information indicating multiple procedures to be performed in the work to be performed.

[0043] The display control unit 410 is realized mainly by cooperation between the processor 101 and the communication unit 106. The display control unit 410 generates display data for displaying the determination result and transmits the generated display data to the UI terminal 60. As a result, a determination result of correctness or incorrectness, such as that shown in FIG. 2, for example, is displayed to the worker 82.

[0044] The transmission unit 411 is mainly realized by the communication unit 106. The transmission unit 411 transmits result data indicating the result of the determination to the host device 70. The transmission unit 411 transmits messages in a format such as Extensible Markup Language (XML) or JavaScript Object Notation (JSON) using a protocol such as Transmission Control Protocol (TCP) or Message Queueing Telemetry Transport (MQTT) to the host device 70, which is a relational database. The host device 70 accumulates the result data and uses the result data for trend analysis, traceability, process control, quality control, or management of the operator 82. The transmission unit 411 may also transmit other data to the host device 70 together with the result data. Examples of the other data to be transmitted include an image on which the determination was made, the serial number of the workpiece, time information, information indicating the operator 82, and torque value or torque waveform data during tightening collected by a sensor built into the tool 10.

[0045] The display control unit 410 and the transmission unit 411 each correspond to an example of an output unit that outputs output information indicating the result of the determination made by the determination unit.

[0046] The learning device 50 also includes a registration unit 51 that registers learning data 501 , a learning unit 53 that learns a model from the learning data 501 , and a reception unit 54 that receives setting information 502 .

[0047] The registration unit 51 is mainly realized by the processor 101. The registration unit 51 registers procedures as shown in FIG. 11 based on the procedures to be performed in a task and their order indicated by the setting information 502. For example, the image "IMG001" is associated with identification information indicating that it was taken during procedure a. As a result, the learning data 501 becomes data in which a plurality of records are accumulated, each of which associates an image with the time the image was taken, the image clarity, the distance estimated from the image, and the procedure in which the image was taken.

[0048] Furthermore, the registration unit 51 may automatically sort the images in the training data 501 by determining the similarity between the images using a machine learning technique such as deep learning or by extracting feature points. If the number of sortings is set, the accuracy of the automatic sorting will be improved. Note that the machine learning technique and feature point extraction for automatic sorting are performed as annotations, unlike classification in the operational phase. These annotations make it possible to detect errors in identification information before training, which can contribute to improving the accuracy of training.

[0049] The registration unit 51 may be realized as a UI by the input unit 104 and the output unit 105, and may have a function of allowing the user to check and edit the contents of the training data 501. It is expected that the user's confirmation will enable the deletion of unnecessary images during model training, thereby improving the training accuracy.

[0050] The learning unit 53 is mainly realized by the processor 101. The learning unit 53 learns various models from the learning data 501. For example, as shown in FIG. 12, the learning unit 53 uses deep learning to learn a classification model that classifies images into one of multiple groups corresponding to each procedure that constitutes a work. Each group corresponding to a procedure may include two or more subgroups. Since the classification model classifies images into one of procedures performed at different parts of the workpiece, it can also be said to be a model that identifies the shooting position. In order to improve the generalization performance of the classification model, the learning unit 53 may generate images with changed brightness, rotated images, and enlarged and cropped images from each image, and then train the classification model.

[0051] 13, the learning unit 53 learns, by deep learning, a tip portion judgment model that specifies judgment targets related to the tip portion 12. The judgment targets related to the tip portion 12 include at least one of the presence or absence of the tip portion 12, the type of the tip portion 12, and the presence or absence of an abnormality in the tip portion 12. The abnormality in the tip portion 12 includes wear and shaft wobble of the tip portion 12.

[0052] For example, the learning unit 53 extracts a portion of the image that shows the tip 12, and creates a model that performs image statistical processing to determine whether the tip 12 is present. Alternatively, the image statistical processing may involve calculating feature amounts for determining the type of tip 12 from the portion that shows the tip 12. The feature amounts are obtained, for example, by focusing on the contour of the edge of the tip 12 or on a portion of the image with a large color difference, and converting the focused contour or portion into data. The calculated feature amounts are stored as a model that indicates a good product. The learning unit 53 also generates a model that utilizes anomaly detection learning. The anomaly detection learning model regards collected images as good products, and learns good product image features using a method such as a convolutional neural network (CNN) using all of these images.

[0053] Furthermore, when one procedure is performed, images are continuously captured. This series of images indicates changes in the position and posture of the tool 10, and indicates the length of time it took to perform the procedure. The learning unit 53 then learns a model that identifies these physical states from the series of images. In detail, the learning unit 53 extracts feature points from the images and learns a portion of the image to be focused on as a model in order to obtain the speed by dividing the movement distance of the feature points from the series of images by the interval between captures.

[0054] 14, if an affine transformation or a mapping transformation is performed so that the feature points of an image overlap with the feature points of a predetermined reference image, the attitude of the tool 10 when the reference image is captured can be used as a reference attitude, and the attitude of the tool 10 when the image was captured can be estimated from the distance between the feature points. For this reason, the learning unit 53 learns the reference image, which serves as a reference for the attitude of the tool 10, as a model for obtaining the attitude of the tool 10.

[0055] As shown in FIG. 15 , the learning unit 53 treats a series of images corresponding to each step as time-series data, calculates standard values ​​for the movement speed of the tool 10, the change in posture, and the length of time required to perform the step, and defines them as a model. Here, the posture of the tool 10 is not an absolute value in the workspace where the worker is present, but a relative value to the posture of the tool 10 when the reference image was captured. Similarly, the movement speed is not an absolute value in the workspace, but a value that appears as the movement of feature points on the image. The learning unit 53 corresponds to an example of a learning means that learns a model that classifies images into one of multiple groups, including groups corresponding to each step, from data generated by a generating means. The learning unit 53 also corresponds to an example of a learning means that learns a model based on data generated by a generating means, and corresponds to an example of a learning means that learns a model that identifies a tip-related determination target from images included in the learning data. Furthermore, the learning unit 53 corresponds to an example of a learning means that learns, from images included in the learning data, a model that identifies the values ​​of physical states, which are at least one of the tool's movement speed and posture change relative to the work object, and at least one of the length of time required for the work or each of the multiple steps that make up the work.

[0056] Returning to Fig. 4, the reception unit 54 is mainly realized by the cooperation of the input unit 104 and the output unit 105. The reception unit 54 receives setting information 502 input by the user and provides the setting information 502 to the registration unit 51 and the reception unit 49 of the processing device 40. The setting information 502 includes information related to learning to be included in the learning data 501 and information used when making a judgment by the task judgment unit 48. The reception unit 54 corresponds to an example of a reception means that receives setting information indicating a plurality of procedures to be performed in a task.

[0057] Next, the processing executed in the work determination system 1000 will be described in detail using Figures 16 to 22. Note that the flowcharts described below are schematic to make the processing flow easier to understand, and the processing flow may be changed as desired.

[0058] 16 shows a flow of a preparation process in which the processing device 40 generates learning data 501 in the preparation phase. In this preparation process, the tool 10 and the processing device 40 establish communication with each other (step S1). Specifically, in the case of a one-to-one direct wired connection, the connected tool 10 is identified by a connection port of the processing device 40, and communication is established. In the case of a connection via a network such as a LAN, a communication-enabled state is established by an identifier on the network such as an IP (Internet Protocol) address. In the case of a wireless connection, communication pairing is performed between the communication unit 16 of the tool 10 and the communication unit 106 of the processing device 40, and a transition to a communication-enabled state is made.

[0059] Next, the photographing unit 11 of the tool 10 photographs an image, the communication unit 16 transmits the photographed image (step S2), and the image acquiring unit 42 of the processing device 40 acquires the image (step S3). The sharpness calculating unit 43 determines whether the sharpness calculated by the sharpness calculating unit 43 for this image exceeds a predetermined threshold value (step S4). If it is determined that the sharpness does not exceed the threshold value (step S4; No), the processes from step S2 onwards are repeated.

[0060] On the other hand, if it is determined that the sharpness exceeds the threshold (step S4; Yes), the distance estimation unit 44 estimates the distance between the tool 10 and the workpiece from the image (step S5). Specifically, the relationship between the feature points and the distance is learned in advance from image data prepared as learning data, and the distance estimation unit 44 estimates the distance based on the learned relationship. Then, the distance acquisition unit 45 acquires distance information indicating the estimated distance (step S6).

[0061] Next, distance acquisition unit 45 determines whether the distance is within a predetermined range (step S7). This range corresponds to an example of a first range. If it is determined that the distance is not within the predetermined range (step S7; No), the processes from step S2 onward are repeated. On the other hand, if it is determined that the distance is within the predetermined range (step S7; Yes), distance acquisition unit 45 stores the image in buffer 46 (step S8).

[0062] In most cases, when working with the tool 10, the distance between the tool 10 and the workpiece falls within a certain range. Therefore, work is not performed at a distance significantly outside this range, and even if the work is performed, it will be considered to be an unusual work and will ultimately have to be redone. Therefore, it can be said that an image taken at a distance outside a certain range is not the result of normal work. If the distance is not normal, it can be determined that the work is abnormal without even identifying the work position, and the work may be excluded from the targets for judgment by the work judgment unit 48, and may ultimately be excluded from the targets for learning.

[0063] Next, the generation unit 47 determines whether a trigger indicating the completion of a procedure has occurred (step S9). Specifically, it determines whether a trigger corresponding to the last procedure constituting the work has occurred. Whether the trigger is the last trigger or another trigger may be identified by a user input indicating the completion of the standard work, the absence of a next trigger for a certain period of time after the trigger is received, or the interruption of image transmission due to the tool 10 transitioning to a power-off state. If it is determined that a trigger has not occurred (step S9: No), the processing from step S2 onward is repeated. On the other hand, if it is determined that a trigger has occurred (step S9; Yes), the generation unit 47 determines whether images with distance variations have been stored in the buffer 46 (step S10). The presence or absence of variation may be determined, for example, by variance or by whether the number of images with different distances exceeds a threshold. The generation unit 47 may make a positive determination in step S10 when two or more images with different distances have been stored in the buffer 46. The generation unit 47 may also determine whether images with distance variations have been stored for each procedure constituting the work.

[0064] If it is determined that images with distance variations are not stored in the buffer 46 (step S10; No), the processes from step S2 onwards are repeated. In this case, the worker 81 may be notified to perform the same procedure or work again. On the other hand, if it is determined that images with distance variations are stored in the buffer 46 (step S10; Yes), a preparation trigger process that is activated by a trigger is executed (step S11).

[0065] 17, in the preparation trigger process, the receiving unit 54 of the learning device 50 receives setting information 502 related to the learning data 501 (step S111). Specifically, the setting information 502 indicating the position and order of the procedure completed in response to the trigger is received.

[0066] Next, the generation unit 47 of the processing device 40 selects two or more images taken at different distances from the buffer 46 (step S112). For example, among the distances associated with the images in the buffer 46, images having the minimum value, maximum value, average value, median value, value corresponding to the quartile, value determined based on the standard deviation, and value closest to a pre-specified value are selected. The method for selecting images is not limited to this, and any selection method may be used as long as there is variation in the distances.

[0067] Next, based on the setting information 502, the registration unit 51 of the learning device 50 registers information in the learning data 501 in a format that enables identification of images (step S113). This completes the learning data 501 as shown in FIG. 11, and preparation of the learning data 501 is complete. Thereafter, the processing by the task classification system 1000 returns from the preparation trigger processing of FIG. 17 to the preparation processing of FIG. 16, and following the preparation trigger processing of step S11, the learning processing is executed (step S12). The above flow for generating the learning data 501 in the preparation processing corresponds to an example of a data generation method.

[0068] 18, in the learning process, the receiving unit 54 receives setting information 502 related to learning (step S121). For example, the setting information 502 sets whether or not to generate a model (described later), whether or not to perform preprocessing on an image when learning the model, and parameters for the preprocessing.

[0069] Next, the learning unit 53 performs preprocessing required for each model on the images of the learning data 501 in accordance with the setting information 502 (step S122). For example, in order to improve the generalization performance of each model, that is, the performance of maintaining the determination accuracy when the working environment changes, the learning unit 53 increases the amount of learning data 501 by changing the brightness, rotating, enlarging, reducing, and cropping the images of the learning data 501.

[0070] Next, the learning unit 53 learns a classification model for determining the work position and procedure from the learning data 501, as shown in FIG. 12 (step S123). Furthermore, the learning unit 53 learns a model for identifying a determination target for the tip portion 12, as shown in FIG. 13 (step S124). Furthermore, the learning unit 53 learns a model for identifying a physical state from a series of images, as shown in FIG. 15 (step S125). Thereafter, the processing by the work determination system 1000 returns from the learning processing in FIG. 18 to the preparation processing in FIG. 16, and the preparation processing ends. This completes preparations for starting the operation phase.

[0071] Next, the operation processing executed in the operation phase will be described. In the operation processing, steps S21 to S28, which are equivalent to steps S2 to S9 of the preparation processing, are executed, as shown in Fig. 19. However, in step S28, it is determined whether any of the triggers corresponding to each procedure has occurred. If it is determined in step S28 that a trigger has occurred (step S28; Yes), the work determination unit 48 executes operation trigger processing (step S29).

[0072] In the operation trigger process, as shown in Fig. 20, the work determination unit 48 selects two or more images taken at different distances from the buffer 46 (step S31). This step S31 may be executed in the same manner as step S112 in Fig. 17, but the image selection method may be different from that of step S112.

[0073] Next, the work determination unit 48 determines whether or not a setting has been made to estimate the tool attitude by referring to the setting information 502 (step S32). If it is determined that a setting has not been made to estimate the tool attitude (step S32; No), the work determination unit 48 proceeds to step S37. On the other hand, if it is determined that a setting has been made to estimate the tool attitude (step S32; Yes), the work determination unit 48 performs preprocessing of the image as necessary, extracts feature points in the image (step S33), and estimates the attitude of the tool 10 based on the extracted feature points as shown in FIG. 14 (step S34).

[0074] Then, the work determination unit 48 determines whether the estimated tool posture is within a predetermined posture range (step S35). If it is determined that the tool posture is not within the predetermined posture range (step S35; No), the display control unit 410 notifies the operator 82 of an abnormality via the UI terminal 60 (step S36). Specifically, an error message such as that shown in FIG. 2 is displayed.

[0075] The image acquisition unit 42 corresponds to an example of an image acquisition means that acquires a reference image captured by an imaging means when the tool's attitude relative to the work object is equal to the reference attitude, and the work judgment unit 48 corresponds to an example of a judgment means that estimates the tool's attitude relative to the object by comparing the actual image with the reference image and judges whether the estimated attitude is within a predetermined attitude range, and corresponds to an example of a judgment means that extracts feature points from each of the actual image and the reference image and estimates the tool's attitude based on the movement distance of the feature points when performing a mapping transformation between the actual image and the reference image. Here, the reference image may be an image captured during a corresponding procedure in the preparation phase, or may be another image captured in a reference attitude.

[0076] On the other hand, if it is determined that the tool posture is within the predetermined posture range (step S35; Yes), the work determination unit 48 executes a determination process (step S37). In the determination process, as shown in FIG. 21, preprocessing is performed on the image as needed (step S41). Examples of preprocessing include processing that uses statistical information of the image, such as image contrast and brightness, and, if a portion of the image to be focused on can be specified, changing the image to one that has been cropped. Execution of preprocessing is expected to improve the accuracy of determination.

[0077] Next, as shown in FIG. 12, the work determination unit 48 uses a classification model to identify the procedure for which the image was captured (step S42) and determines whether the identified procedure matches the contents of the setting information 502 (step S43). Here, inference based on deep learning classification learning generally derives a probability value or score value that applies to each registered part of the workpiece. The procedure corresponding to the part with the highest applicable probability value or score value is identified. If the probability value or score value is low for all registered parts, it should be determined to be a recognition error, and a threshold for determining a recognition error may be set. If the probability value or score value for all registered parts does not exceed the threshold, it is determined to be a judgment error, and the procedure may be determined again for the next best image. The work determination unit 48 corresponds to an example of a determination means that determines the procedure for which the actual image was captured and determines whether the determined procedure corresponds to the procedure indicated by the setting information.

[0078] If it is determined that the identified procedure does not match the contents of the setting information 502 (step S43; No), the display control unit 410 notifies the operator 82 of an abnormality (step S48). On the other hand, if it is determined that the identified procedure matches the contents of the setting information 502 (step S43; Yes), the work determination unit 48 determines the determination target related to the tip portion 12 using the tip portion determination model as shown in FIG. 13 (step S44). Then, the work determination unit 48 determines whether the determination target is appropriate (step S45). Specifically, it determines whether the tip portion 12 is shown in the image and attached to the tool 10, whether the type of the tip portion 12 shown in the image matches the type specified in the setting information 502, and whether an abnormality has occurred in the tip portion 12. The work determination unit 48 corresponds to an example of a determination means that identifies the determination target shown in the implementation image using the learned model. Then, if it is determined that the determination target is inappropriate (step S45; No), the process proceeds to step S48. The task determination unit 48 corresponds to an example of a determination means that determines the procedure in which the execution image was captured and determines whether the determined procedure corresponds to the procedure indicated by the setting information.

[0079] On the other hand, if it is determined that the object to be determined is appropriate (step S45; Yes), the work determination unit 48 uses the physical state identification model to identify the physical state as shown in FIG. 15 (step S46) and determines whether the physical state is appropriate or not (step S47). Specifically, it determines whether the movement speed of the tool 10, the change in posture, and the length of time required for the procedure fall within a range determined based on the standard work performed in the preparation phase. This range corresponds to an example of a second range. The work determination unit 48 corresponds to an example of a determination means that determines whether the value of the physical state of the performed work identified using the model from the series of performed images acquired by the image acquisition means is within a predetermined second range. If it is determined that the physical state is inappropriate (step S47; No), the process proceeds to step S48.

[0080] On the other hand, if it is determined that the physical state is appropriate (step S47; Yes), the display control unit 410 notifies the UI terminal 60 that the procedure is normal (step S49). This allows the worker 82 to confirm that there is no problem with the procedure that has been performed immediately after the procedure is completed.

[0081] Next, the work determination unit 48 determines whether or not it is set that a visual inspection should be performed (step S410). If it is determined that it is set that a visual inspection should be performed (step S410; Yes), the work determination unit 48 executes a visual inspection process (step S411).

[0082] 22, in the visual inspection process, the work determination unit 48 reads images before and after the occurrence of a trigger indicating the end of a procedure from the buffer 46 (step S51). Here, to ensure that images before and after the execution of the procedure are read, the first and last images in the buffer 46 may be read.

[0083] Next, the operation judgment unit 48 performs preprocessing for the visual inspection on the read image (step S52). For example, the contrast and brightness values ​​of the images in the learning data 501 may be stored in the preparation phase, and a statistical value such as the average, maximum, or minimum value of those values ​​may be selected, and the contrast and brightness of the image for the visual inspection may be matched to that statistical value.

[0084] Next, the work judgment unit 48 inspects the appearance before and after performing the procedure (step S53). For example, the work judgment unit 48 judges the presence or absence of scratches using a previously trained anomaly detection model. As the anomaly detection model, for example, a good / bad learning model may be used that learns the characteristics of good products using good product data acquired when the work is performed normally as learning data 501.

[0085] Next, the work determination unit 48 outputs the result of the appearance inspection (step S54). Specifically, the display control unit 410 displays pass / fail as the result of the appearance inspection on the UI terminal 60. Thereafter, the processing by the processing device 40 returns from the appearance inspection processing of FIG. 22 to the determination processing of FIG. 21, and then returns to the operation trigger processing of FIG. 20, and the operation trigger processing ends. Then, when the operation trigger processing ends, the processing by the processing device 40 returns to the operation processing of FIG. 19.

[0086] Following the operation trigger process of step S29, the display control unit 14 determines whether any abnormality has been detected in the operation trigger process (step S210). If it is determined that an abnormality has been detected (step S210; Yes), the processes from step S21 onwards are repeated. This allows a determination to be made on the work that will continue to be performed by the worker 82 who has been notified of the abnormality. On the other hand, if it is determined that no abnormality has been detected (step S210; No), the display control unit 14 displays a message that the procedure has been properly performed (step S211). For example, as illustrated in FIG. 2, a representative image from among the images processed in the operation trigger process is displayed, along with a check mark. Thereafter, the processes from step S21 onwards are repeated. The above flow for making a determination regarding work in the operation phase corresponds to an example of a work determination method.

[0087] As described above, in the task determination system 1000, a task-related determination is made each time a trigger corresponding to one of the steps constituting the task occurs. Therefore, when a trigger corresponding to one step occurs, the result of the determination related to that step is provided to the worker 82 before a trigger corresponding to the next step occurs. This eliminates the need to redo the entire task even if the worker 82 performs a task that includes an incorrect step, thereby reducing the burden on the worker 82. Ultimately, tasks that are managed by photographing can be carried out efficiently.

[0088] Furthermore, in both the preparation phase and the operation phase, if the sharpness does not exceed the threshold, the image having that sharpness is excluded from being stored in the buffer 46. Therefore, images with low sharpness are excluded from learning targets and are also excluded from targets for work-related judgment. This allows a model trained on high-sharpness images to perform work-related judgments on high-sharpness images. This is expected to improve learning accuracy and judgment accuracy. In particular, work judgments can be made with high accuracy even when using an inexpensive camera that is prone to blurry images.

[0089] Additionally, in both the preparation phase and the operation phase, images in which the distance between the tool and the workpiece is not within a certain range are excluded from being stored in the buffer 46. Therefore, a model trained from images taken at the distance expected during the work is used to make judgments about the work on images taken at such a distance. This is expected to improve the learning accuracy and judgment accuracy.

[0090] Furthermore, in the preparation phase, the model was trained from multiple images with varying distances. Therefore, in addition to images of the tool 10 in contact with the workpiece, images of the tool 10 approaching and moving away from the workpiece are also used as training targets. This is expected to improve the accuracy of work-related judgments, even in cases where, for example, the tool 10 vibrates significantly while the switch 13 is in the ON state, making it difficult to obtain high-definition images when the tool 10 is in contact with the workpiece. Furthermore, by including images taken when the tool 10 is not in contact with the workpiece as training targets, it is possible to easily obtain the training data required for each work location in the operation phase, thereby reducing the burden of preparatory work in the preparation phase. Ultimately, this reduces the burden of preparatory work required for managing work by taking photographs.

[0091] Furthermore, the work judgment unit 48 does not directly judge the captured image, but judges the image temporarily stored in the buffer 46. It is desirable that the image to be judged has high clarity. Therefore, images obtained after selection of clarity suitable for judgment are buffered. This limits the images to be judged to those with high clarity, thereby improving the judgment accuracy. Furthermore, by excluding images with low clarity from the images to be judged by the work judgment unit 48, a reduction in the calculation load, a reduction in the storage area for images to be buffered, and easier reuse of images are achieved.

[0092] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0093] For example, while the description has focused on an example in which a model is learned by a learning device 50, the present invention is not limited to this example, and the learning device 50 may be omitted from the configuration of the work judgment system 1000, as shown in FIG. 23. The generation unit 47 may generate judgment data 500 including images stored in the buffer 46, and the work judgment unit 48 may make a judgment regarding the work by comparing the images included in the judgment data 500 with images captured in the operation phase. Furthermore, in the configuration of FIG. 23, the processing device 40 may perform functions equivalent to those of the learning device 50 according to the above embodiment.

[0094] Also, although an example in which the distance between the tool 10 and the workpiece is estimated from an image has been described, the present invention is not limited to this. For example, as shown in Fig. 24, instead of the distance estimation unit 44, the tool 10 may be provided with a distance measurement unit 17 that measures the distance using ultrasonic waves or a laser, and the distance acquisition unit 45 may acquire distance information indicating the measured distance from the distance measurement unit 17. However, when the distance is obtained from a captured image, it is possible to reduce the size of the tool 10 and avoid high system costs.

[0095] Furthermore, while the above description has focused on an example in which a task includes multiple steps, the present invention is not limited to this example, and the task may substantially correspond to one step. When the task substantially corresponds to one step, the visual inspection is performed by comparing images taken before and after the task. However, the visual inspection may also be performed by comparing images taken before and after a task that includes multiple steps. The task judgment unit 48 corresponds to an example of a judgment means that inspects the appearance of the object of the task by comparing images taken before and after the task or before and after a single step.

[0096] The photographing unit 11 may also be built into the tool 10.

[0097] In addition, although an example has been described in which the work procedures in the preparation phase are performed in the same order as in the operation phase, this is not limited to this. The order of the procedures performed in the preparation phase may be different from that in the operation phase. In addition, a case has been described in which the preparation trigger process is performed when the entire work including multiple procedures is completed, but this is not limited to this. The preparation trigger process may be performed each time a trigger corresponding to each procedure occurs. Then, the learning process may be performed after the preparation trigger process corresponding to the last procedure is completed. In addition, the preparation trigger process and the learning process may be performed each time a trigger corresponding to each procedure occurs, and the model may be updated each time a procedure is performed.

[0098] In addition, although an example has been described in which the generation of learning data in the preparation phase and the determination of tasks in the operation phase are performed by a single processing device 40, the present invention is not limited to this. The generation of learning data in the preparation phase and the determination of tasks in the operation phase may be performed by different processing devices 40.

[0099] In the example of FIG. 12, the classification by the classification model classifies images into one of the groups corresponding to each of the steps constituting the work, but this is not limited to this. For example, the learning unit 53 may learn a classification model that classifies images into one of multiple groups including groups corresponding to each of the steps constituting the work and an abnormal group that does not correspond to any of the steps. The group that does not correspond to a step may be omitted, or there may be multiple groups. By learning the classification model from training data that includes a group corresponding to an abnormal step, it is possible to classify not only normal steps but also abnormal steps with high accuracy, and to determine what kind of abnormality has occurred.

[0100] Furthermore, while the above description focuses on examples in which work is performed by human workers, the present invention is not limited to this. The workers in the above embodiments may be replaced with robots. Even when work is performed by a robot, the time required to redo the work due to rework can be reduced.

[0101] In the above embodiment, the explanation has been centered on an example in which each step corresponds to one screw tightening operation, but this is not limiting. One step may correspond to multiple screw tightening operations. Furthermore, not all steps included in a job may be subject to judgment by the processing device 40.

[0102] In the above embodiment, an example has been described in which images are extracted by the sharpness calculation unit 43 and the distance acquisition unit 45 from multiple images acquired by the image acquisition unit 42, stored in the buffer 46, and read out by the generation unit 47. However, the present invention is not limited to this. In the preparation phase, the generation unit 47 may perform the functions of the sharpness calculation unit 43, the distance acquisition unit 45, and the buffer 46. In other words, the generation unit 47 may extract images based on the sharpness and the distance.

[0103] The generation unit 47 corresponds to an example of a generation means that extracts two or more standard images from the plurality of standard images, the distances of which are different from each other and acquired by the distance acquisition means, as extracted images, and generates data from the extracted images, and the processing device 40 having the generation unit 47 corresponds to an example of a data generation device that generates data for making a judgment regarding the performed work from the performed images obtained by photographing the performed work. Also, the generation unit 47 corresponds to an example of a generation means that extracts, from the plurality of standard images, extracted images whose clarity, which indicates the degree to which the subject is clearly captured, is greater than a threshold, and corresponds to an example of a generation means that extracts, from the plurality of standard images, two or more standard images whose distances acquired by the distance acquisition means are within a predetermined first range.

[0104] Furthermore, in the configuration of FIG. 23 , the processing device 40 may perform functions equivalent to those of the learning device 50 according to the above-described embodiment. For example, in the above-described embodiment, the registration unit 51 of the learning device 50 assigned, to the image, identification information indicating the procedure being performed when the image was captured. Alternatively, the generation unit 47 may assign, to the image, identification information for identifying the procedure. Here, because a trigger signal is generated at a timing corresponding to each procedure, the identification information for identifying the procedure can also be considered identification information for identifying the trigger signal. The generation unit 47 corresponds to an example of a generation means for generating data including an extracted image to which identification information for identifying the trigger signal received when the extracted image was captured during standard work has been assigned.

[0105] The functions of the processing device 40 and learning device 50 according to the above-described embodiment can be realized by dedicated hardware or by an ordinary computer system.

[0106] For example, program P1 can be stored and distributed on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), or an MO (Magneto-Optical disk), and by installing program P1 on a computer, a device that executes the above-mentioned processing can be configured.

[0107] Furthermore, the program P1 may be stored in a disk device of a server device on a communication network such as the Internet, and may be downloaded to a computer by superimposing it on a carrier wave, for example.

[0108] The above process can also be achieved by starting and executing the program P1 while transferring it via a network such as the Internet.

[0109] Furthermore, the above-described processing can also be achieved by executing all or part of program P1 on a server device, and executing program P1 while the computer sends and receives information about the processing via a communications network.

[0110] In addition, if the above functions are realized by an operating system (OS) or by a collaboration between the OS and an application, only the parts other than the OS may be stored on a medium and distributed, or may be downloaded to a computer.

[0111] Furthermore, the means for realizing the functions of the processing device 40 and the learning device 50 are not limited to software, and some or all of them may be realized by dedicated hardware or circuits.

[0112] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure. [Industrial Applicability]

[0113] The present disclosure is suitable for managing operations carried out in a factory. [Explanation of symbols]

[0114] 10 Tool, 11 Photography unit, 12 Tip, 13 Switch, 14 Wire, 15 Trigger monitoring unit, 16 Communication unit, 17 Distance measurement unit, 20, 21, 30, 31 Work, 22a, 22b, 22c, 22d, 32a, 32b, 32c, 32d Screw, 40 Processing device, 41 Trigger receiving unit, 42 Image acquisition unit, 43 Sharpness calculation unit, 44 Distance estimation unit, 45 Distance acquisition unit, 46 Buffer, 47 Generation unit, 48 Work judgment unit, 49 Reception unit, 410 Display control unit, 411 Transmission unit, 50 Learning device, 51 Registration unit, 53 Learning unit, 54 Reception unit, 60 UI terminal, 70 Upper device, 81, 82 Worker, 90 Arithmetic unit, 101 Processor, 102 Main memory unit, 103 Auxiliary memory unit, 104 input unit, 105 output unit, 106 communication unit, 107 internal bus, 500 judgment data, 501 learning data, 502 setting information, 1000 work judgment system, P1 program.

Claims

1. A data generation device that generates data for making a judgment regarding an execution work from an execution image obtained by photographing the execution work, an image acquisition means for acquiring a plurality of standard images obtained by photographing the state of use of the tool in a standard work performed as a standard for the actual work by means of an image acquisition means attached to the tool; a distance acquisition means for acquiring distance information indicating a distance between the tool shown in each of the standard images acquired by the image acquisition means and a work object in the standard work in which the tool is used; a generating means for extracting, as extracted images, two or more standard images having different distances acquired by the distance acquiring means from the plurality of standard images, and generating the data from the extracted images; A data generating device comprising:

2. the generating means extracts the extracted image having a clarity that indicates the degree to which a subject is clearly captured, the clarity being greater than a threshold value, from the plurality of standard images; The data generating device according to claim 1 .

3. a distance estimation means for estimating the distance from the standard image, the distance acquisition means acquires the distance information indicating the distance estimated by the distance estimation means. The data generating device according to claim 1 .

4. The generating means extracts, from the plurality of standard images, two or more standard images for which the distance acquired by the distance acquiring means is within a predetermined first range. The data generating device according to claim 1 .

5. further comprising a trigger receiving means for receiving a trigger signal; The standard work and the actual work each include a plurality of procedures, the trigger receiving means receives the trigger signal generated at a timing corresponding to each of the steps; the generating means generates the data including the extracted image to which identification information for identifying the trigger signal received when the extracted image was photographed during the standard work has been added. The data generating device according to claim 1 .

6. The data generating device according to claim 5 ; a learning means for learning a model for classifying images into one of a plurality of groups including groups corresponding to each of the procedures from the data generated by the generating means; a determination means for making a determination regarding the work to be performed using the model; an output means for outputting information indicating the result of the determination by the determination means; Equipped with The image acquisition means acquires the actual image, The determination means determines the procedure in which the implementation image was captured. Work judgment system.

7. The system further includes a receiving unit configured to receive setting information indicating a plurality of procedures to be performed in the work to be performed, The determination means determines the procedure in which the implementation image was captured, and determines whether the determined procedure corresponds to the procedure indicated by the setting information. The work determination system according to claim 6 .

8. A data generating device according to any one of claims 1 to 5; a learning means for learning a model based on the data generated by the generating means; a determination means for making a determination regarding the work to be performed using the model; an output means for outputting output information indicating the result of the determination by the determination means; Equipped with The image acquisition means acquires the actual image, The determination means determines the execution work based on the execution image. Work judgment system.

9. The tool is used in combination with one tip portion selected from a plurality of types of tip portions that act on the workpiece, the learning means learns the model that identifies a determination target including at least one of the presence or absence of the tip portion, the type of the tip portion, and the presence or absence of an abnormality in the tip portion from the extracted image included in the data; The determination means identifies the determination target appearing in the implementation image using the model. The work determination system according to claim 8 .

10. the learning means learns the model that identifies values ​​of physical states, which are at least one of a moving speed and a change in posture of the tool relative to the work object, and a length of time required for the work or for each of a plurality of steps that make up the work, from the extracted image included in the data; The image acquisition means acquires a series of the implementation images; The determination means determines whether or not a value of the physical state of the performed work identified using the model from the series of performed images acquired by the image acquisition means is within a predetermined second range. The work determination system according to claim 8 .

11. A data generation method for generating data for making a judgment regarding an execution work from an execution image obtained by photographing the execution work, comprising: The image acquisition means acquires a plurality of standard images obtained by photographing the usage state of the tool in a standard work performed as a standard for the performed work by using an imaging means attached to the tool, a distance acquisition means for acquiring distance information indicating a distance between the tool shown in each of the standard images acquired by the image acquisition means and a work object in the standard work in which the tool is used; a generating means for extracting, from the plurality of standard images, two or more standard images for which the distances acquired by the distance acquiring means are different from each other as extracted images, and generating the data from the extracted images; A data generation method comprising:

12. a computer that generates data for making a judgment about an execution work from an execution image obtained by photographing the execution work, an image acquisition means for acquiring a plurality of standard images obtained by photographing the state of use of the tool in a standard work performed as a standard for the actual work using an imaging means attached to the tool; a distance acquisition means for acquiring distance information indicating the distance between the tool shown in each of the standard images acquired by the image acquisition means and a work object in the standard work in which the tool is used; a generating means for extracting, as extracted images, two or more standard images having different distances acquired by the distance acquiring means from the plurality of standard images, and generating the data from the extracted images; A program to function as a

Images