Inspection device, learning device, inspection method, learning device production method, and program

Abstract

To solve the problem in which: conventional inspection devices cannot determine whether a work including two or more procedures is correctly performed by shooting the work.SOLUTION: An inspection device 2 comprises: an inspection moving image acquisition unit 221 for acquiring an inspection moving image, which is a moving image in which inspection target work being work of the inspection target including two or more procedures is shot; a determination unit 222 for determining whether the procedures of the inspection target work for the inspection moving image is correct using teacher data based on a normal moving image, which is a moving image in which normal work including two or more procedures is shot; and an output unit 23 for outputting determination result information on determination results of the determination unit 222. The inspection device makes it possible to determine whether the work is being correctly performed by shooting work including two or more procedures.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an inspection device that uses video of a series of operations having two or more steps to inspect whether the operations are appropriate and outputs the inspection results. [Background technology]

[0002] Conventionally, there has been an inspection method in which a captured image of an object to be inspected is subjected to image processing to detect defects (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6218094 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the prior art, it was not possible to determine whether a task having two or more steps was being performed correctly according to the scheduled steps by photographing the task. [Means for solving the problem]

[0005] The inspection device of the first invention is an inspection device that includes an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more steps, is being performed, a judgment unit that judges whether the procedure of the inspection target work for the inspection video is correct or not using teacher data based on a normal video, which is a video captured while a normal work, which is a correct work having two or more steps, is being performed, and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

[0006] With this configuration, by capturing an image of a task having two or more steps, it is possible to determine whether the task is being performed correctly according to the scheduled steps.

[0007] Furthermore, the inspection device of the second invention is an inspection device in which, compared to the first invention, the inspection video acquisition unit sequentially acquires inspection videos while the work to be inspected is being performed, and the judgment unit judges whether work corresponding to the procedure to be currently performed is being performed for one or more fields contained in the inspection videos sequentially acquired by the inspection video acquisition unit, by using the teacher data among the teacher data that corresponds to the procedure to be currently performed.

[0008] With this configuration, by capturing an image of a task having two or more steps, it is possible to determine in real time whether the task is being performed correctly, and to notify the worker of the result of the determination.

[0009] In addition, the inspection device of the third invention, compared to the first or second inventions, further includes a current procedure storage unit in which a current procedure identifier that identifies the procedure currently being inspected among two or more procedures is stored, the teacher data has one field corresponding to each of the two or more procedures, and the judgment unit includes an inspection image acquisition means for acquiring one field of the inspection video acquired by the inspection video acquisition unit, and a judgment means for using the one field acquired by the inspection image acquisition means and the one field corresponding to the current procedure identifier to determine whether the procedure corresponding to the one field acquired by the inspection image acquisition means is the procedure identified by the current procedure identifier.

[0010] With this configuration, by capturing an image of a task having two or more steps, it is possible to quickly determine whether the task is being performed correctly.

[0011] In addition, the inspection device of the fourth invention is an inspection device in which, compared to the third invention, the teacher data is a set of a field corresponding to each of two or more procedures and a procedure identifier identifying each procedure, and the inspection device further includes a learning device storage unit in which a learning device constructed by performing a learning process on the two or more teacher data using a machine learning algorithm is stored, and the judgment means performs a machine learning prediction process using the one field and the learning device acquired by the inspection image acquisition means, acquires a procedure identifier corresponding to the one field, and judges whether the procedure identifier matches the current procedure identifier.

[0012] With this configuration, by capturing an image of a task having two or more steps, it is possible to quickly and flexibly determine whether the task is being performed correctly.

[0013] Furthermore, compared to the third invention, the inspection device of the fifth invention is an inspection device in which the judgment means calculates a similarity between a field corresponding to each of two or more procedures and a field acquired by the inspection image acquisition means, and uses the two or more similarities to judge whether the procedure corresponding to the field acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

[0014] With this configuration, by capturing an image of a task having two or more steps, it is possible to quickly determine whether the task is being performed correctly.

[0015] In addition, the inspection device of the sixth invention, compared to the first or second inventions, further includes a current procedure storage unit in which a current procedure identifier that identifies the procedure currently being inspected among the two or more procedures is stored, the teacher data has two or more fields corresponding to each of the two or more procedures, and the judgment unit includes an inspection image acquisition means for acquiring two or more fields of the inspection video acquired by the inspection video acquisition unit, and a judgment means for using the two or more fields acquired by the inspection image acquisition means and two or more fields corresponding to the current procedure identifier to determine whether the procedure corresponding to the two or more fields acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

[0016] With this configuration, by capturing an image of a task having two or more steps, it is possible to accurately and flexibly determine whether the task is being performed correctly.

[0017] In addition, the inspection device of the seventh invention is an inspection device in which, compared to the sixth invention, the teacher data is a set of two fields corresponding to two or more procedures and a procedure identifier identifying each procedure, and the inspection device further includes a learning device storage unit in which a learning device constructed by performing a learning process on the two or more teacher data using a machine learning algorithm is stored, and the judgment means performs a prediction process using a machine learning algorithm using the two or more fields and the learning device acquired by the inspection image acquisition means, acquires a procedure identifier corresponding to the two or more fields, and judges whether the procedure identifier matches the current procedure identifier.

[0018] With this configuration, by capturing an image of a task having two or more steps, it is possible to accurately and flexibly determine whether the task is being performed correctly.

[0019] Furthermore, the inspection device of the eighth invention is an inspection device according to any one of the first to seventh inventions, wherein the judgment unit acquires one or more fields which are a portion of fields from the inspection video that satisfy predetermined conditions, and uses the one or more fields and teacher data to judge whether the procedure of the work to be inspected for the inspection video is correct or not.

[0020] With this configuration, by capturing an image of a task having two or more steps, it is possible to quickly and flexibly determine whether the task is being performed correctly.

[0021] In addition, the inspection device of the ninth invention is an inspection device according to any one of the first to eighth inventions, in which the judgment unit performs a machine learning prediction process using a second learning device to determine whether or not each field in the inspection video will be used as a field for determining whether or not it is a correct procedure, obtains one or more fields corresponding to the prediction result that they will be used, and uses the one or more fields and teacher data to judge whether or not the procedure of the work to be inspected for the inspection video is correct.

[0022] With this configuration, by capturing an image of a task having two or more steps, it is possible to determine with high accuracy whether the task is being performed correctly.

[0023] In addition, the inspection device of the tenth invention is an inspection device according to any one of the first to ninth inventions, further comprising an equipment-related information acquisition unit that acquires equipment-related information, which is information input into the device for the work to be inspected, or information generated within the device, or information output from the device, and the judgment unit uses teacher data based on normal equipment-related information, which is equipment-related information when normal work is performed, and normal videos to judge whether the procedure of the work to be inspected is correct for the inspection video and the equipment-related information.

[0024] With this configuration, it is possible to determine whether or not the work has been performed correctly by also using the device-related information when the work has been performed on the device used for the work.

[0025] In addition, the inspection device of the eleventh invention, compared to the tenth invention, further comprises a normal equipment-related information storage unit in which normal equipment-related information, which is equipment-related information when normal work is performed, is stored, and the judgment unit comprises a first judgment means for judging whether or not the procedure of the work to be inspected is correct using the equipment-related information and the normal equipment-related information acquired by the equipment-related information acquisition unit, and a second judgment means for judging whether or not the procedure of the work to be inspected for the inspection video is correct using teacher data based on the normal video when the judgment result of the first judgment means is that the procedure is correct.

[0026] With this configuration, after making a judgment using the device-related information, only if the judgment is normal is a judgment made using the inspection image, so that the normality of the work can be judged accurately and efficiently.

[0027] In addition, the inspection device of the twelfth invention, compared to the tenth invention, further comprises a normal equipment-related information storage unit in which normal equipment-related information, which is equipment-related information when normal work is performed, is stored, and the judgment unit comprises a second judgment means for using teacher data based on the normal video to judge whether the procedure of the work to be inspected for the inspection video is correct, and a first judgment means for judging whether the procedure of the work to be inspected is correct, using the equipment-related information and normal equipment-related information acquired by the equipment-related information acquisition unit if the judgment result of the second judgment means is correct.

[0028] With this configuration, after making a judgment using the inspection image, only if the judgment is normal is a judgment made using the apparatus-related information, so that the normality of the work can be judged accurately and efficiently.

[0029] In addition, the inspection device of the thirteenth invention is an inspection device in which, compared to the tenth invention, the teacher data is a set of one field corresponding to each of two or more procedures, normal equipment-related information, and a procedure identifier identifying each procedure, and the inspection device further includes a learning device storage unit in which a learning device constructed by performing a learning process on the two or more teacher data using a machine learning algorithm is stored, and the judgment unit acquires one field of the inspection video acquired by the inspection video acquisition unit, and uses the one field, the equipment-related information acquired by the equipment-related information acquisition unit, and the learning device to acquire a procedure identifier identifying the procedure corresponding to the one field by machine learning prediction processing, and judges whether the procedure identifier matches the current procedure identifier.

[0030] With this configuration, it is possible to use the machine learning algorithm to determine whether or not the work has been performed correctly, also using device-related information when the work is performed on the device used for the work.

[0031] In addition, the learning device of the fourteenth invention is a learning device comprising: a learning storage unit in which a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, is being performed; a teacher data acquisition unit that acquires one or more fields that satisfy predetermined conditions from the normal video for each of the two or more steps, and acquires two or more teacher data that are pairs of a step identifier that identifies the step corresponding to the field and the field; and a learning unit that performs machine learning learning processing on the two or more teacher data, acquires a learning device, and accumulates the learning device.

[0032] This configuration provides a learning machine that can be used by a testing device.

[0033] In addition, the learning device of the fifteenth invention is a learning device comprising: a teacher data storage unit in which one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, being performed and normal equipment-related information, which is equipment-related information when the normal operation is performed; a teacher data acquisition unit that acquires, for each of the two or more steps, two or more teacher data, which are a combination of one or more fields that satisfy predetermined conditions, normal equipment-related information, and a procedure identifier, from the normal video; and a learning unit that performs a machine learning learning process using the two or more teacher data, acquires a learning device, and accumulates the learning device.

[0034] This configuration provides a learning machine that can be used by a testing device. Effect of the Invention

[0035] According to the inspection device of the present invention, by photographing an operation having two or more steps, it is possible to determine whether the operation is being performed correctly. [Brief description of the drawings]

[0036] [Figure 1] 1 is a conceptual diagram of an inspection system A according to the first embodiment. [Diagram 2] Block diagram of the inspection system A [Diagram 3]A flowchart for explaining an example of the operation of the inspection device 2. [Figure 4] A flowchart illustrating an example of the field acquisition process. [Diagram 5] 11 is a flowchart illustrating an example of the first determination process. [Figure 6] 11 is a flowchart illustrating an example of the second determination process. [Figure 7] 11 is a flowchart illustrating an example of the third determination process. [Figure 8] 11 is a flowchart illustrating an example of the fourth determination process. [Figure 9] FIG. 11 is a block diagram of a learning device 3 according to the second embodiment. [Figure 10] 1 is a flowchart illustrating a first operation example of the learning device 3. [Figure 11] 11 is a flowchart illustrating a second operation example of the learning device 3. [Figure 12] A flowchart illustrating an example of the field acquisition process. [Figure 13] A diagram showing the field management table [Figure 14] 1 is a conceptual diagram of an inspection system B according to a third embodiment. [Figure 15] Block diagram of the same inspection system B [Figure 16] A flowchart for explaining an example of the operation of the inspection device 4 [Figure 17] 11 is a flowchart illustrating a first example of the determination process. [Figure 18] 11 is a flowchart illustrating a second example of the determination process. [Figure 19] 11 is a flowchart illustrating a third example of the determination process. [Figure 20] A flowchart for explaining the operation of the fourth example of the determination process. [Figure 21] 5 is a flowchart illustrating a fifth example of the determination process. [Figure 22] A flowchart illustrating an example of the first determination process. [Diagram 23] Block diagram of a learning device 5 according to the fourth embodiment. [Figure 24] 1 is a flowchart illustrating a first operation example of the learning device 5. [Diagram 25] 11 is a flowchart illustrating a second operation example of the learning device 5. [Figure 26] Overview of the computer system according to the above embodiment. [Figure 27] Block diagram of the computer system DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Hereinafter, an embodiment of an inspection device and the like will be described with reference to the drawings. In the embodiments, components with the same reference numerals perform similar operations, so repeated description may be omitted.

[0038] (Embodiment 1) In this embodiment, an inspection device is described that uses an inspection video captured during a series of tasks having two or more steps and normative teacher data to inspect whether the tasks are being performed according to the planned steps and outputs the inspection results.

[0039] In addition, in this embodiment, an inspection device will be described that sequentially receives inspection videos during work and inspects in real time whether the work is being performed according to a scheduled procedure.

[0040] In the present embodiment, an inspection device that performs inspection using one or more fields corresponding to each of two or more procedures will be described.

[0041] In addition, in this embodiment, an inspection device that uses a machine learning algorithm to inspect whether or not work is being performed according to a scheduled procedure will be described.

[0042] In addition, in this embodiment, an inspection device that uses a machine learning algorithm when selecting a field to be used for inspection from an inspection video of a series of operations having two or more steps will be described.

[0043] 1 is a conceptual diagram of an inspection system A according to the present embodiment. The inspection system A includes one or more cameras 1 and an inspection device 2.

[0044] Camera 1 captures a series of tasks having two or more steps and obtains an inspection video. An inspection video is a video of the object of inspection. Inspection is an inspection to check whether a series of tasks having two or more steps is being performed according to the planned steps. Note that camera 1 may be fixed, or may be placed in a moving environment, such as being installed on a moving object.

[0045] The inspection device 2 is a device that inspects the inspection video captured by the camera 1, inspects whether a series of tasks having two or more steps is performed according to the scheduled steps, and outputs the inspection results. The inspection device 2 is, for example, a so-called personal computer, a tablet terminal, a smartphone, a server, etc., and the type does not matter. The server is, for example, a cloud server, an ASP server, and the type does not matter.

[0046] The camera 1 and the inspection device 2 may be integrated into one device. Such a device is, for example, a tablet terminal or a smartphone equipped with a camera.

[0047] 2 is a block diagram of the inspection system A in this embodiment. The inspection device 2 includes a storage unit 21, a processing unit 22, and an output unit .

[0048] The storage unit 21 includes a teacher data storage unit 211, a learning device storage unit 212, a threshold storage unit 213, and a current procedure storage unit 214. The processing unit 22 includes an inspection video acquisition unit 221 and a determination unit 222. The determination unit 222 includes an inspection image acquisition means 2221 and a determination means 2222.

[0049] Camera 1 captures an image of the work to be inspected, which is work to be inspected that has two or more steps, being performed, and obtains a video. The video is usually an inspection video. The work to be inspected may be a video including or excluding a worker. It is preferable that camera 1 is installed in a location where the results of the work in each step can be well captured. The steps may be called steps, etc. The worker may be a person or a device. The device is, for example, a robot or a personal computer. The robot may be an industrial robot used in the manufacture of some kind of item, or a humanoid robot, etc., and the type is not important.

[0050] Various types of information are stored in the storage unit 21 constituting the inspection device 2. The various types of information include, for example, teacher data (to be described later), a learning device (to be described later), a threshold value (to be described later), a current procedure identifier (to be described later), and procedure information (to be described later).

[0051] Procedure information is information that identifies two or more procedures that make up a task. Procedure information has, for example, two or more procedure identifiers in the order of the procedures. The procedure identifier is information that identifies a procedure. The procedure identifier is, for example, an ID or a procedure name. Procedure information has, for example, two or more pairs of information indicating the order of the procedures and the procedure identifier. For example, procedure information for the task of making ramen is, for example, "Pour in the soup, pour in the noodles, pour in the toppings (such as green onions)" or "(1, pour in the soup) (2, pour in the noodles) (3, pour in the toppings)."

[0052] The teacher data storage unit 211 stores two or more teacher data. The teacher data is information used for comparison with part or all of the inspection video. The teacher data is usually data based on a normal video (referred to as a positive example as appropriate). A normal video is a video captured while normal work is being performed. A normal work is a correct work that has two or more steps. Data based on the normal video is information acquired using the normal video. The teacher data includes, for example, one or more fields that make up the normal video.

[0053] The teacher data storage unit 211 usually has one or more teacher data for each procedure. The teacher data for each procedure usually corresponds to a procedure identifier. The teacher data is, for example, one field (which may be called a representative field) that represents the procedure. The teacher data is, for example, two or more fields that correspond to each procedure. The teacher data is, for example, a normal scene video that corresponds to each procedure. The normal scene video is a video that has been shot from the beginning to the end of one procedure. The teacher data may have a procedure identifier. The teacher data may be associated with the procedure identifier. Note that being associated with a procedure identifier may be considered to mean having a procedure identifier.

[0054] The one or more teacher data may include data based on an abnormal video, which is a video capturing an abnormal task (referred to as a negative example as appropriate). A negative example is information acquired using an abnormal video. A negative example includes, for example, one or more fields that constitute an abnormal video. A negative example may include, for example, a flag indicating that the task is abnormal. A negative example may have, for example, a procedure identifier, or may be associated with the procedure identifier.

[0055] In addition, when the worker is a device such as a robot or a personal computer, the normal video may be information in which a simulation device (not shown) simulates normal operations performed by the device and records a video of the results. In addition, the abnormal video may be information in which a simulation device (not shown) simulates abnormal operations that the device may perform and records a video of the results. In addition, since the simulation device that simulates operations that the device C may perform and outputs the simulation results is a publicly known technology, detailed explanation will be omitted.

[0056] The learning device storage unit 212 stores a first learning device in addition to the teacher data storage unit 211 or instead of the teacher data storage unit 211. The learning device storage unit 212 may also store a second learning device.

[0057] The first learning device is a learning device used when determining whether or not the procedure of the inspection target work for the inspection video is correct. The first learning device is a learning device configured by performing a learning process on two or more pieces of teacher data using a machine learning algorithm. The learning device may be called a classifier, a model, or the like.

[0058] A first learning device may exist for each procedure in the learning device storage unit 212. In other words, each of the two or more first learning devices may be associated with a procedure identifier.

[0059] The second learning device is a learning device for selecting a field to be used for determining whether or not the procedure of the work to be inspected is correct from the inspection video. The second learning device is, for example, a learning device configured by performing a learning process on one or more inspection selection positive example fields and one or more inspection selection negative example fields using a machine learning algorithm. The inspection selection positive example field is a field used for determining whether or not the procedure of the work to be inspected is correct. The inspection selection positive example field is a field used for inspection. The inspection selection negative example field is a field not used for inspection. The inspection selection positive example field and the inspection selection negative example field are, for example, fields manually selected from normal videos and / or abnormal videos.

[0060] Moreover, it is preferable that the first learning device and the second learning device are learning devices acquired by a learning device 3, which will be described later.

[0061] The machine learning algorithm may be, for example, deep learning, random forest, decision tree, SVM, etc. However, the machine learning algorithm is not limited. The fact that the machine learning algorithm is not limited applies to both the learning process and the prediction process.

[0062] The threshold value storage unit 213 stores a threshold value related to the similarity with the teacher data. For example, when the similarity between the field acquired from the inspection video and the teacher data of one procedure is equal to or greater than the threshold value, it is determined that the procedure corresponding to the field acquired from the inspection video is the one procedure. The threshold value may be stored for each procedure. In other words, each of the one or more threshold values ​​may be associated with a procedure identifier.

[0063] The current procedure storage unit 214 stores a current procedure identifier. The current procedure identifier is information that identifies the procedure currently being inspected among two or more procedures. For example, if the procedure information is "Pour soup, Pour noodles, Pour toppings", the current procedure identifier is, for example, "Pour soup". Also, if the procedure information is, for example, "(1, Pour soup) (2, Pour noodles) (3, Pour toppings)", the current procedure identifier is, for example, "1" or "Pour soup".

[0064] The processing unit 22 performs various types of processing. The various types of processing are, for example, processing performed by an inspection video acquisition unit 221 and a determination unit 222.

[0065] The inspection video acquisition unit 221 acquires the inspection video. The inspection video is a video captured while the work to be inspected is being performed. The work to be inspected is a task that is the subject of inspection and has two or more steps. The work to be inspected is, for example, the task of preparing a dish (e.g., ramen noodles) or the task of manufacturing a product. The inspection video acquisition unit 221 receives a video from, for example, the camera 1 capturing the scene where the work to be inspected is being performed. The inspection video acquisition unit 221 reads out, for example, inspection images stored in a recording medium (not shown) by the camera 1 capturing the scene where the work to be inspected is being performed.

[0066] It is preferable that the inspection video acquisition unit 221 sequentially acquires the inspection video in a situation where the work to be inspected is being performed. In other words, it is preferable that the inspection video acquisition unit 221 sequentially acquires the inspection video in real time while the work to be inspected is being performed.

[0067] The judgment unit 222 judges whether or not the procedure of the work to be inspected for the inspection video is correct using teacher data based on the normal video. Note that a normal video is a video captured of a normal work being performed. Also, a normal work is a correct work having two or more procedures. An inspection video is a video of the work to be inspected. A work to be inspected is a work that is the subject of inspection.

[0068] The judgment unit 222 judges whether or not an operation corresponding to a procedure to be performed is currently being performed for one or more fields of the inspection video sequentially acquired by the inspection video acquisition unit 221, by using the teacher data corresponding to the procedure to be performed currently among the teacher data. Note that a field is a still image in the video, and may be called a frame or a still image. Also, the teacher data corresponding to the procedure to be performed is teacher data that is paired with a current procedure identifier.

[0069] The judgment unit 222 acquires one or more fields, which are a portion of fields that satisfy predetermined conditions from the inspection video, and uses the one or more fields and the teacher data to judge whether the procedure of the work to be inspected for the inspection video is correct.

[0070] In other words, the judgment unit 222 performs a judgment process to judge whether or not the procedure of the work to be inspected for the inspection video is correct by using, for example, one or more fields contained in the inspection video and teacher data. In addition, as a pre-processing of the judgment process, the judgment unit 222 usually performs a field acquisition process to acquire some fields that satisfy a predetermined condition from the inspection video.

[0071] The predetermined condition is, for example, that a predetermined object (e.g., a ramen bowl) is recognized by image recognition. The predetermined condition is, for example, that a machine learning prediction process is used to determine whether or not a procedure is correct.

[0072] The inspection image acquisition means 2221 constituting the judgment unit 222 performs field acquisition processing.

[0073] The inspection image acquisition means 2221 acquires, for example, one field of the inspection video acquired by the inspection video acquisition unit 221. It is preferable that the one field is a field that satisfies a predetermined condition. The predetermined condition is, for example, that a person's hand does not overlap with the frame of the ramen bowl, and that the bowl exists within the field.

[0074] The inspection image acquisition means 2221 acquires, for example, two or more fields of the inspection video acquired by the inspection video acquisition unit 221. It is preferable that the two or more fields are fields that satisfy a predetermined condition. The predetermined condition is, for example, that a person's hand does not overlap with the frame of the ramen bowl, and that the bowl exists within the field.

[0075] The field acquisition process performed by the inspection image acquisition means 2221 is, for example, one of the following two. (1) Image recognition method

[0076] The inspection image acquisition means 2221 uses image recognition to determine whether or not each field in the inspection video contains a predetermined object (for example, a ramen bowl). Then, the inspection image acquisition means 2221 acquires two or more fields containing the predetermined object. Note that two or more fields refer to one or more fields for each of two or more steps. Also, since such image recognition technology is a publicly known technology, detailed description will be omitted.

[0077] The inspection image acquisition means 2221 determines, for example, whether or not a predetermined object (for example, a ramen bowl) is present by image recognition, regardless of the procedure.

[0078] For example, the inspection image acquisition means 2221 determines, for each step, whether or not each field of the moving image of the scene corresponding to the step identified by the current step identifier has a different object by image recognition. (2) Machine learning method

[0079] The inspection image acquisition means 2221, for example, performs a usability determination for each field in the inspection video using machine learning prediction processing, and acquires one or more fields that correspond to the usability determination result of the usability determination that the field can be used.

[0080] The usability determination is a process of performing a machine learning prediction process using a second learning device to obtain a prediction result of whether or not to use the procedure. The second learning device is a learning device for determining whether or not to use the procedure as a field for determining whether or not the procedure is correct.

[0081] The test image acquisition means 2221, for example, uses a second learning device, regardless of the procedure, to make a usability determination for each field in the test video, and acquires one or more fields that correspond to the usability determination result of the usability determination.

[0082] The inspection image acquisition means 2221, for example, uses a second learning device that pairs with the current procedure identifier for each procedure to determine whether each field in the inspection video can be used, and acquires one or more fields that correspond to the result of the usability determination that the field can be used.

[0083] As a pre-processing of (1) or (2) above, partial videos (scenes) for each step may be detected from the inspection video. Scene detection means detection of scene divisions. The inspection image acquisition means 2221, for example, judges whether a field in the video satisfies a predetermined scene division condition. If the scene division condition is satisfied, the field is judged to be the start field of the scene. Then, the field immediately before the start field of the scene is the end field of the previous scene.

[0084] The scene division conditions are, for example, that the similarity between the immediately preceding field and the current field of interest is equal to or less than a threshold, that the magnitude of a motion vector obtained based on the difference between the immediately preceding field and the current field of interest is equal to or greater than a threshold, and that an object corresponding to a scene division can be recognized by image recognition processing. Note that various known techniques can be used for scene division in a moving image.

[0085] The judgment means 2222 performs judgment processing. The judgment processing is processing for judging whether or not the procedure of the work to be inspected for the inspection video is correct. In the judgment processing, the judgment means 2222 may use one field acquired by the inspection image acquisition means 2221 for each procedure, or may use two or more fields.

[0086] In other words, the judgment means 2222, for example, uses a field acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the current procedure identifier to determine whether the procedure corresponding to the field acquired by the inspection image acquisition means 2221 is a procedure identified by the current procedure identifier.

[0087] In addition, the judgment means 2222 uses the two or more fields acquired by the inspection image acquisition means 2221 and the two or more fields corresponding to the current procedure identifier to judge whether the procedure corresponding to the two or more fields acquired by the inspection image acquisition means 2221 is a procedure identified by the current procedure identifier.

[0088] The determining means 2222 performs the following two examples of determination processing. (1) Machine learning

[0089] The determination means 2222 performs a machine learning prediction process using, for example, the one field acquired by the inspection image acquisition means 2221 and the first learning device, and acquires a procedure identifier corresponding to the one field. Next, the determination means 2222 determines whether the acquired procedure identifier matches the current procedure identifier in the current procedure storage unit 214. If they do not match, the determination means 2222 acquires an error judgment result. Note that the error judgment result is a judgment result that the procedures are different. Note that the first learning device is given to a module that performs machine learning prediction process together with the one field, and is a learning device for acquiring a procedure identifier by executing the module. Also, if it is determined that the prediction process matches the current procedure identifier, the determination means 2222 rewrites the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the next procedure. Note that the determination means 2222 acquires the procedure identifier of the next procedure by referring to the procedure information in the storage unit 21. If it is determined that the current procedure identifier matches the current procedure identifier, it means that the procedure identified by the current procedure identifier has been performed.

[0090] The determination means 2222 performs a prediction process by a machine learning algorithm using two or more fields acquired by the inspection image acquisition means 2221 and a first learning device, and acquires a procedure identifier corresponding to the two or more fields. Next, the determination means 2222 judges whether the acquired procedure identifier matches the current procedure identifier. If they do not match, the determination means 2222 acquires an error judgment result. Note that the first learning device is given to a module that performs a machine learning prediction process together with two or more fields, and is a learning device for acquiring a procedure identifier by executing the module. Also, when it is determined that the result of the prediction process matches the current procedure identifier, the determination means 2222 rewrites the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the next procedure. Note that the determination means 2222 refers to the procedure information in the storage unit 21 and acquires the procedure identifier of the next procedure from the current procedure identifier. Also, the module may be an execution module, a function, a method, or the like.

[0091] It is preferable that the judgment means 2222 does not regard an error as occurring when, after judging that a procedure identified by the current procedure identifier has been performed, a procedure identifier corresponding to the same procedure is acquired.

[0092] Furthermore, when the determination means 2222 determines that the procedure identified by the current procedure identifier has been performed, it overwrites the current procedure identifier with the procedure identifier of the next procedure. (2) When using field similarity

[0093] The determination means 2222 calculates the similarity between the fields (teacher data) corresponding to each of two or more procedures and one field for each procedure acquired by the inspection image acquisition means 2221, and uses the similarity to determine whether or not the procedure corresponding to the one field acquired by the inspection image acquisition means 2221 is a procedure identified by the current procedure identifier. If it is a procedure identified by the current procedure identifier, it means that the procedure has been performed. Note that the process of calculating the similarity between two fields is a known technique, so a detailed description will be omitted.

[0094] The determination means 2222, for example, calculates a similarity between one field of the current procedure acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the current procedure identifier. The determination means 2222 also calculates a similarity between one field of the current procedure acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the procedure identifiers of each procedure after the current procedure. The determination means 2222 then compares the two or more calculated similarities and determines whether the similarity with the field (teacher data) corresponding to the current procedure identifier is the maximum. If it is not the maximum, the determination means 2222 obtains a determination result that the procedures are different.

[0095] If the similarity with the field (teacher data) corresponding to the current procedure identifier is equal to or smaller than the threshold, the determination means 2222 acquires an error determination result.

[0096] It is preferable that the judgment means 2222 does not regard an error as occurring when, after judging that a procedure identified by the current procedure identifier has been performed, a procedure identifier corresponding to the same procedure is acquired.

[0097] Furthermore, when the determination means 2222 determines that the procedure identified by the current procedure identifier has been performed, it overwrites the current procedure identifier with the procedure identifier of the next procedure.

[0098] The output unit 23 outputs judgment result information regarding the judgment result of the judgment unit 222. The judgment result information is, for example, "normal" or "error." Here, output is a concept including display on a display, projection using a projector, printing on a printer, sound output, transmission to an external device, storage on a recording medium, delivery of the processing result to another processing device, another program, etc.

[0099] The storage unit 21, the teacher data storage unit 211, the learning device storage unit 212, the threshold storage unit 213, and the current procedure storage unit 214 are preferably non-volatile recording media, but may also be realized as volatile recording media.

[0100] There is no restriction on the process by which information is stored in the storage unit 21, etc. For example, information may be stored in the storage unit 21, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 21, etc., or information inputted via an input device may be stored in the storage unit 21, etc.

[0101] The processing unit 22, the inspection video acquisition unit 221, the determination unit 222, the inspection image acquisition means 2221, and the determination means 2222 can usually be realized by a processor, a memory, etc. The processing procedure of the processing unit 22, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit). The processor may be a CPU, an MPU, a GPU, etc.

[0102] The output unit 23 may be considered to include or not include an output device such as a display, a speaker, etc. The output unit 23 may be realized by driver software for an output device, or a combination of driver software for an output device and an output device, etc.

[0103] Next, an example of the operation of the inspection system A will be described. First, the operation of the camera 1 will be described. The camera 1 captures an image of an inspection target work having two or more steps being performed, and transmits the acquired image to the inspection device 2 in real time. The camera 1 may store the acquired image in a recording medium accessible by the inspection device 2.

[0104] Next, an example of the operation of the inspection device 2 will be described with reference to the flowchart of FIG.

[0105] (Step S301) The processing unit 22 judges whether or not to start an inspection. If the inspection is to be started, the processing proceeds to step S302, and if the inspection is not to be started, the processing returns to step S301.

[0106] (Step S302) The processing unit 22 assigns 1 to a counter i.

[0107] (Step S303) The processing unit 22 refers to the procedure information in the storage unit 21 and judges whether or not the i-th procedure exists. If the i-th procedure exists, the process proceeds to step S304, and if the i-th procedure does not exist, the process proceeds to step S308. Note that, at this point, it is preferable for the processing unit 22 to rewrite the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the i-th procedure.

[0108] (Step S304) The inspection moving image acquiring unit 221 performs a field acquiring process. An example of the field acquiring process will be described with reference to the flowchart of FIG.

[0109] (Step S305) The determination means 2222 performs a determination process. An example of the determination process will be described with reference to the flowcharts of FIGS.

[0110] (Step S306) The processing unit 22 judges whether the result of the judgment process in step S305 is “normal” or “error.” If it is “normal,” the process proceeds to step S307, and if it is “error,” the process proceeds to step S309.

[0111] (Step S307) The processing unit 22 increments the counter i by 1. The process returns to step S303.

[0112] (Step S308) The determination unit 222 assigns the value "normal" to the determination result information. Then, the process proceeds to step S310.

[0113] (Step S309) The determination unit 222 assigns the value "error" to the determination result information.

[0114] (Step S310) The output unit 23 outputs the determination result information. Return to step S301.

[0115] In the flowchart of FIG. 3, the process ends when the power is turned off or an interrupt occurs to end the process.

[0116] Furthermore, in the flowchart of FIG. 3, the current procedure identifier in current procedure storage unit 214 is updated to the next procedure identifier in accordance with the increment of i.

[0117] Next, an example of the field acquisition process in step S304 will be described with reference to the flowchart in FIG.

[0118] (Step S401) The inspection video acquisition unit 221 judges whether or not a field of the candidate inspection target has been acquired. For example, the inspection video acquisition unit 221 judges whether or not a field of the candidate inspection target has been received from the camera 1. If a field of the candidate inspection target has been acquired, the process proceeds to step S402, and if a field of the candidate inspection target has not been acquired, the process returns to step S401. Note that it is preferable for the inspection video acquisition unit 221 to, for example, sequentially receive fields constituting the inspection video captured by the camera 1. It is also preferable for the inspection video acquisition unit 221 to, for example, sequentially read out fields of the video in a recording medium (not shown).

[0119] (Step S402) The inspection image acquisition means 2221 judges whether or not the field acquired in step S401 is a field to be inspected. If it is a field to be inspected, the process proceeds to step S403, and if it is not a field to be inspected, the process returns to step S401.

[0120] The inspection image acquisition means 2221 makes the determination, for example, by the above-mentioned field acquisition process (1) a method using image recognition or (2) a method using machine learning. When acquiring two or more fields, the inspection image acquisition means 2221 determines, for example, a field having a predetermined relationship with the first acquired field as the field to be inspected. Note that the field having the predetermined relationship is, for example, one or more consecutive fields within a time period within a threshold from the field first determined to be the field to be inspected. The field having the predetermined relationship is, for example, a field having a similarity to the field first determined to be the field to be inspected that is equal to or greater than a threshold.

[0121] (Step S403) The inspection image acquiring means 2221 acquires the one field acquired in step S401, and temporarily stores it in a buffer (not shown).

[0122] (Step S404) The inspection image acquisition means 2221 judges whether or not to acquire two or more fields. If two or more fields are to be acquired, the process proceeds to step S405, and if two or more fields are not to be acquired, the process returns to the upper process. Note that whether or not to acquire two or more fields may be determined in advance.

[0123] (Step S405) The inspection image acquisition means 2221 judges whether or not to end the acquisition of fields. If it is to end, it returns to the upper process, and if it is not to end, it returns to step S401. Note that, for example, in one procedure, the number of fields to be acquired is determined, and when this number is reached, the inspection image acquisition means 2221 judges to end the acquisition of fields.

[0124] Next, an example of the first determination process in step S305 will be described with reference to the flowchart in Fig. 5. The first determination process is a case in which machine learning for multi-value classification is used.

[0125] (Step S501) The determination means 2222 acquires a first learning device from the learning device storage unit 212.

[0126] (Step S502) The determination means 2222 acquires one or more fields acquired in step S304.

[0127] (Step S503) The determination means 2222 provides the first learning device acquired in step S501 and the one or more fields acquired in step S502 to a module that performs machine learning prediction processing, and executes the module.

[0128] (Step S504) The determination means 2222 acquires the procedure identifier which is the execution result in step S503.

[0129] (Step S505) The judgment means 2222 judges whether or not the procedure identifier acquired in step S504 matches the current procedure identifier in the current procedure storage unit 214. If they match, the process proceeds to step S506, and if they do not match, the process proceeds to step S507.

[0130] (Step S506) The determination means 2222 assigns "normal" to the determination result information, and returns to the upper level process.

[0131] (Step S507) The determination means 2222 assigns "error" to the determination result information, and returns to the upper level process.

[0132] In the flowchart of FIG. 5, machine learning of multi-value classification was performed using a first learning device. That is, in the flowchart of FIG. 5, the determination means 2222 used a common first learning device that is not related to the procedure. However, the determination means 2222 may acquire a first learning device that is paired with the current procedure identifier. Such a first learning device is a learning device for binary classification (determines whether or not the procedure corresponds). Then, when the determination means 2222 obtains a prediction result that corresponds to the procedure identified by the current procedure identifier as a result of the prediction process, it assigns "normal" to the determination result information, and when the determination means 2222 obtains a prediction result that does not correspond to the procedure identified by the current procedure identifier, it assigns "error" to the determination result information.

[0133] Next, an example of the second determination process in step S305 will be described with reference to the flowchart in Fig. 6. The second determination process is a case where machine learning for binary classification is used. Note that in the flowchart in Fig. 6, the description of the same steps as in the flowchart in Fig. 5 will be omitted.

[0134] (Step S601) The decision means 2222 assigns 1 to a counter i.

[0135] (Step S602) The judgment means 2222 judges whether or not a first learning device corresponding to the i-th procedure exists in the learning device storage unit 212. If the i-th first learning device exists, the process proceeds to step S603, and if the i-th first learning device does not exist, the process proceeds to step S608. Each of the two or more first learning devices corresponds to a procedure identifier.

[0136] (Step S603) The determination means 2222 acquires the i-th first learning device from the learning device storage unit 212.

[0137] (Step S604) The determination means 2222 acquires one or more fields acquired in step S304.

[0138] (Step S605) The determination means 2222 provides the i-th first learning device and one or more fields to a module that performs prediction processing for binary classification in machine learning.

[0139] (Step S606) The judgment means 2222 executes the module. Then, the judgment means 2222 acquires the identification result (yes or no) of whether one or more fields correspond to the i-th procedure and the score, and temporarily stores them in a buffer (not shown). Note that the score is a score output by the module that performs the prediction process, and is a value indicating the likelihood of the identification result.

[0140] (Step S607) The decision means 2222 increments the counter i by 1. The process returns to step S602.

[0141] (Step S608) The judgment means 2222 acquires a procedure identifier using the classification result and the score temporarily accumulated in step S605. The judgment means 2222 acquires a procedure identifier corresponding to the first learning device with the highest score, which corresponds to the classification result that is applicable. The process proceeds to step S505.

[0142] Next, an example of the third determination process in step S305 will be described with reference to the flowchart in Fig. 7. The third determination process is a method that uses the similarity between the training data of the current procedure and the field to be inspected.

[0143] (Step S701) The determination means 2222 acquires teacher data paired with the current procedure identifier from the teacher data storage unit 211. Note that such teacher data is a field constituting a normal moving image.

[0144] (Step S702) The decision means 2222 assigns 1 to a counter i.

[0145] (Step S703) The decision means 2222 decides whether or not the i-th field to be checked exists. If the i-th field to be checked exists, the process proceeds to step S704, and if not, the process proceeds to step S706.

[0146] (Step S704) The determination means 2222 calculates the similarity between the training data acquired in step S701 and the i-th field to be inspected, and temporarily stores the similarity in a buffer (not shown).

[0147] (Step S705) The determination means 2222 increments the counter i by 1. The process returns to step S703.

[0148] (Step S706) The judgment means 2222 judges whether or not the one or more similarities acquired in step S704 satisfy a predetermined similarity condition. If the similarity condition is satisfied, the process proceeds to step S707, and if the similarity condition is not satisfied, the process proceeds to step S708. Note that the similarity condition is, for example, that the representative similarity is equal to or greater than a threshold value. Note that, if there is only one similarity acquired in step S704, the representative similarity is that similarity, and if there are two or more similarities acquired in step S704, the representative similarity is a representative value (for example, an average value, a top value, a median value, a bottom value) of the two or more similarities.

[0149] (Step S707) The determination means 2222 assigns "normal" to the determination result information, and returns to the upper level process.

[0150] (Step S708) The determination means 2222 assigns "error" to the determination result information, and returns to the upper level process.

[0151] Next, an example of the fourth determination process in step S305 will be described with reference to the flowchart in Fig. 8. The fourth determination process is a method that uses the similarity between the training data of each of two or more procedures and the field to be inspected.

[0152] (Step S801) The decision means 2222 assigns 1 to a counter i.

[0153] (Step S802) The judgment means 2222 judges whether or not the i-th procedure to be compared exists. If the i-th procedure exists, the process proceeds to step S803, and if not, the process proceeds to step S809. The i-th procedure to be compared is, for example, the current procedure and the procedures thereafter.

[0154] (Step S803) The determination unit 2222 acquires training data paired with the procedure identifier of the i-th procedure. Note that the training data is, for example, a field constituting a normal video.

[0155] (Step S804) The decision means 2222 assigns 1 to the counter j.

[0156] (Step S805) The decision means 2222 decides whether or not the j-th field to be inspected exists. If the j-th field to be inspected exists, the process proceeds to step S806, and if not, the process proceeds to step S808.

[0157] (Step S806) The determination means 2222 calculates the similarity between the training data acquired in step S803 and the j-th field to be inspected.

[0158] (Step S807) The determination means 2222 increments the counter j by 1. The process returns to step S805.

[0159] (Step S808) The determination means 2222 increments the counter i by 1. The process returns to step S802.

[0160] (Step S809) The judgment means 2222 judges whether the two or more similarities calculated in step S806 satisfy the similarity condition. If the similarity condition is satisfied, the process proceeds to step S810, and if the similarity condition is not satisfied, the process proceeds to step S811. Note that the similarity condition is, for example, that the procedure identifier paired with the training data corresponding to the maximum similarity matches the current procedure identifier. The similarity condition is, for example, that the procedure identifier of the procedure having the largest representative value (for example, average, median, maximum, minimum) of the two or more similarities corresponding to each procedure matches the current procedure identifier.

[0161] (Step S810) The determination means 2222 assigns "normal" to the determination result information, and returns to the upper level process.

[0162] (Step S811) The determination means 2222 assigns "error" to the determination result information, and returns to the upper level process.

[0163] As described above, according to this embodiment, by capturing an image of a task having two or more steps, it is possible to determine whether the task is being performed correctly according to the scheduled steps.

[0164] The process in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software for realizing the inspection device 2 in this embodiment is a program as follows. That is, this program is a program for making a computer function as an inspection video acquisition unit that acquires an inspection video, which is a video taken of an inspection target work, which is an inspection target work having two or more steps, being performed, a judgment unit that judges whether or not the procedure of the inspection target work for the inspection video is correct using teacher data based on a normal video, which is a video taken of a normal work, which is a correct work having two or more steps, being performed, and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

[0165] (Embodiment 2) In this embodiment, a description will be given of a learning device 3 that acquires a learning device used by the inspection device 2. Note that the learning device here is the above-mentioned first learning device or second learning device.

[0166] 9 is a block diagram of the learning device 3 in this embodiment. The learning device 3 includes a learning storage unit 31 and a learning processing unit 32. The learning processing unit 32 includes a teacher data acquisition unit 321 and a learning unit 322.

[0167] Various types of information are stored in the learning storage unit 31. The various types of information are, for example, one or more normal videos, teacher data, a first learning device, and a second learning device. The normal video is a video captured when a normal task, which is a correct task having two or more steps, is being performed.

[0168] The learning storage unit 31 may store two or more pieces of teacher data for configuring the second learning device. That is, such teacher data is, for example, a field selected by the user from a normal video, and is a field (positive example) used in the judgment process of the judgment means 2222. Also, such teacher data is, for example, a field selected by the user from a normal video, and is a field (negative example) not used in the judgment process of the judgment means 2222.

[0169] Each of the two or more teacher data for configuring the second learning device may be associated with a procedure identifier. The teacher data may have a procedure identifier. The teacher data may be a pair of a field selected from a normal video by a user and a procedure identifier input by the user.

[0170] The learning processing unit 32 performs various processes. The various processes are, for example, processes performed by a teacher data acquisition unit 321 and a learning unit 322.

[0171] The teacher data acquisition unit 321 acquires one or more fields that satisfy a predetermined condition from the normal video for each of two or more steps. The teacher data acquisition unit 321 also acquires a step identifier that identifies the step corresponding to the one or more fields. Next, the teacher data acquisition unit 321 acquires two or more pieces of teacher data that are pairs of the acquired one or more fields and step identifiers for each of two or more steps. Note that the one or more fields that make up the teacher data are fields that make up the video captured during the work of the step identified by the paired step identifier.

[0172] The process of acquiring fields by the teacher data acquiring unit 321 may be the same as the field acquiring process performed by the inspection image acquiring means 2221. That is, the teacher data acquiring unit 321 acquires one or more fields to be used for learning from a normal video by, for example, either (1) the method based on image recognition or (2) the method based on machine learning of the inspection image acquiring means 2221 described above.

[0173] The learning unit 322 performs a machine learning learning process on two or more pieces of teacher data, acquires a first learning device, and stores the first learning device. Note that the first learning device may be stored in, for example, the learning storage unit 31, but is not limited thereto.

[0174] The learning unit 322 performs a machine learning learning process on, for example, two or more pieces of teacher data, acquires a first learning device, and accumulates the first learning device. The first learning device is a learning device used in a prediction process that receives a field and outputs a procedure identifier. The first learning device is a learning device used when performing multi-value classification. As described above, any machine learning algorithm may be used.

[0175] For example, the learning unit 322 learns, for each procedure identifier, one or more fields (positive examples) constituting a video of the procedure identified by the procedure identifier and one or more fields (negative examples) not constituting a video of the procedure by using a machine learning learning process, acquires a second learning device, and accumulates the second learning device. Note that the second learning device is accumulated in pairs with the procedure identifier for each procedure identifier. The second learning device is a learning device used for prediction processing that receives a field, determines whether or not the field corresponds to the procedure corresponding to the second learning device, and outputs a score. The second learning device is a learning device used when performing binary classification. Note that, as described above, any machine learning algorithm is used.

[0176] The learning unit 322 performs a machine learning learning process on two or more pieces of teacher data, acquires a second learning device, and accumulates the second learning device. The second learning device may be accumulated in, for example, the learning storage unit 31, but is not limited thereto. The two or more pieces of teacher data are teacher data for constituting the second learning device. The second learning device is a learning device for classifying whether or not to use the procedure in the judgment process for judging the procedure, and is a learning device for performing binary classification.

[0177] The learning unit 322 acquires a second learning device for each procedure, for example, and accumulates the second learning device. In this case, the learning unit 322 provides two or more pieces of training data to a module that performs machine learning learning processing, with one or more pieces of training data paired with a procedure identifier that identifies the procedure as positive examples, and one or more pieces of training data not paired with the procedure identifier that identifies the procedure as negative examples, for each procedure, executes the module, and acquires a second learning device. Note that, as described above, any machine learning algorithm is acceptable.

[0178] The learning storage unit 31 is preferably a non-volatile recording medium, but may also be realized as a volatile recording medium.

[0179] There is no restriction on the process by which information is stored in the learning storage unit 31, etc. For example, information may be stored in the learning storage unit 31, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the learning storage unit 31, etc., or information inputted via an input device may be stored in the learning storage unit 31, etc.

[0180] The learning processing unit 32, the teacher data acquisition unit 321, and the learning unit 322 can usually be realized by a processor, a memory, etc. The processing procedure of the learning processing unit 32, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuit). The processor may be a CPU, MPU, GPU, etc.

[0181] Next, a first operation example of the learning device 3 will be described with reference to the flowchart in Fig. 10. In the flowchart in Fig. 10, a first learning device for multi-value classification is acquired. The first learning device is a learning device that receives one or more fields as input and outputs a procedure identifier.

[0182] (Step S1001) The learning processing unit 32 assigns 1 to a counter i.

[0183] (Step S1002) The learning processing unit 32 judges whether or not the i-th video exists in the learning storage unit 31. If the i-th video exists, the process proceeds to step S1003, and if not, the process proceeds to step S1009. Note that the i-th video is usually the i-th normal video.

[0184] (Step S1003) The teacher data acquiring unit 321 assigns 1 to a counter j.

[0185] (Step S1004) The teacher data acquisition unit 321 judges whether or not the j-th procedure exists. If the j-th procedure exists, the process proceeds to step S1005, and if not, the process proceeds to step S1008.

[0186] (Step S1005) The teacher data acquisition unit 321 performs a field acquisition process corresponding to the j-th procedure. The field acquisition process here is a process for acquiring one or more fields corresponding to the j-th procedure. An example of the field acquisition process here will be described with reference to the flowchart of FIG.

[0187] (Step S1006) The teacher data acquiring unit 321 accumulates the one or more fields acquired in step S1005 in the learning storage unit 31 in association with the procedure identifier of the j-th procedure.

[0188] (Step S1007) The learning processing unit 32 increments the counter j by 1. The process returns to step S1004.

[0189] (Step S1008) The learning processing unit 32 increments the counter i by 1. The process returns to step S1002.

[0190] (Step S1009) The learning unit 322 provides two or more pieces of teacher data stored in the learning storage unit 31 to a machine learning learning processing module, executes the module, and acquires a first learning device.

[0191] (Step S1010) The learning unit 322 stores the first learning device acquired in step S1009, and then ends the process.

[0192] Next, a second operation example of the learning device 3 will be described with reference to the flowchart of FIG. 11. In the flowchart of FIG. 11, a first learning device for binary classification is obtained for each procedure identifier. The first learning device is a learning device that receives one or more fields as input and outputs whether or not the procedure corresponds to the first learning device. In the flowchart of FIG. 11, the description of the same steps as those in the flowchart of FIG. 10 will be omitted.

[0193] (Step S1101) The learning unit 322 assigns 1 to a counter k.

[0194] (Step S1102) Learning unit 322 judges whether or not the kth procedure exists. If the kth procedure exists, the process proceeds to step S1103, and if the kth procedure does not exist, the process ends.

[0195] (Step S1103) The learning unit 322 acquires from the learning storage unit 31 one or more fields (positive examples) that are paired with the procedure identifier of the kth procedure.

[0196] (Step S1104) The learning unit 322 acquires from the learning storage unit 31 one or more fields (negative examples) that are not paired with the procedure identifier of the kth procedure.

[0197] (Step S1105) The learning unit 322 provides one or more positive examples and one or more negative examples to a module that performs a machine learning learning process, executes the module, and acquires a first learning device.

[0198] (Step S1106) The learning unit 322 stores the first learning device acquired in step S1105 in a pair with the procedure identifier of the k-th procedure.

[0199] (Step S1107) The learning unit 322 increments the counter k by 1. The process returns to step S1102.

[0200] Next, an example of the field acquisition process in step S1005 will be described with reference to the flowchart in FIG. (Step S1201) The teacher data acquisition unit 321 assigns X to a counter k. Note that X is the counter value in the i-th moving image to be processed. X is not initialized in the loop processing from S1004 to S1007 in Figs. 10 and 11. The initial value of X is "1". (Step S1202) The teacher data acquisition unit 321 determines whether or not the kth field exists in the moving image to be processed. If the kth field exists, the process proceeds to step S1203, and if the kth field does not exist, the process returns to the upper process. (Step S1203) The teacher data acquisition unit 321 acquires the k-th field of the moving image to be processed. (Step S1204) The teacher data acquisition unit 321 judges whether or not the k-th field is a field to be learned. If it is a field to be learned, the process proceeds to step S1205, and if it is not a field to be learned, the process proceeds to step S1207.

[0201] The method for determining whether the kth field is a field to be learned is (1) the image recognition method or (2) the machine learning method of the inspection image acquisition means 2221 described above, and a detailed explanation will be omitted. (Step S1205) The teacher data acquiring unit 321 temporarily stores the k-th field in a buffer (not shown). (Step S1206) The teacher data acquisition unit 321 judges whether or not to end the acquisition of fields in the j-th procedure. If it is to end, it returns to the upper process, and if it is not to end, it goes to step S1207. If the maximum number of fields to be acquired in the j-th procedure is determined, the teacher data acquisition unit 321 judges to end the acquisition of fields in the j-th procedure when the maximum number of fields is accumulated in step S1205. However, the maximum number of fields to be acquired does not have to be determined. In addition, after accumulating a field to be learned one or more times, if it is determined in step S1204 that the field is not a learning target, it is preferable to determine that the acquisition of fields in the j-th procedure is to end as a transition to the next procedure. (Step S1207) The teacher data acquiring unit 321 increments the counter k by 1. The process returns to step S1202.

[0202] The specific operation of the learning device 3 will now be described.

[0203] Now, assume that the procedure information (1, pour in the soup), (2, pour in the noodles), and (3, pour in the toppings) is stored in the learning storage unit 31 of the learning device 3. In other words, the procedure information is information in which the procedure identifiers "1", "2", and "3" correspond to the character strings "Pour in the soup", "Pour in the noodles", and "Pour in the toppings", which indicate the three procedures that constitute the work of making ramen.

[0204] Furthermore, the learning storage unit 31 of the learning device 3 stores a field management table shown in Fig. 13. The field management table stores fields (still images) that are teacher data for configuring a first learning device. The fields in Fig. 13 are fields acquired by the field acquisition process of the learning processing unit 32. The fields correspond to procedure identifiers. Furthermore, in Fig. 13, "ID" is information that identifies a record.

[0205] In such a situation, the learning unit 322 acquires two or more pieces of teacher data having a pair of a field and a procedure identifier from the field management table shown in FIG.

[0206] Next, the learning unit 322 provides a large amount of teacher data to a machine learning learning processing module, executes the module, and acquires and accumulates a first learning device. The first learning device is a learning device for multi-value classification. The first learning device is a learning device used for prediction processing that receives a field and outputs a procedure identifier. Here, it is assumed that the first learning device is accumulated in the learning device storage unit 212 of the inspection device 2.

[0207] Next, a specific example of the operation of the above-mentioned inspection system A will be described. A conceptual diagram of the inspection system A is shown in Fig. 1. Assume that the work to be inspected is the work of making ramen. Also, assume that the storage unit 21 stores procedure information (1, pour soup) (2, pour noodles) (3, pour toppings) indicating the work of making ramen.

[0208] Also, it is assumed that camera 1 is installed in a position where it can capture the ramen making process (particularly, the inside of the ramen bowl).

[0209] In such a situation, the following two specific examples will be described. Specific Example 1 is a case where the work is performed normally. Specific Example 2 is a case where the work is not performed normally. It is assumed that the storage unit 21 of the inspection device 2 stores the procedure information (1, pour soup) (2, pour noodles) (3, pour toppings). In other words, the normal work is to perform the steps in the order of "pour soup", "pour noodles", and "pour toppings".

[0210] (Example 1) Now, worker A is making ramen. He puts the soup, noodles, and toppings (green onions, boiled eggs, etc.) into the bowl in that order.

[0211] The camera 1 then captures images of the work of the worker A and transmits the captured video to the inspection device 2 in sequence.

[0212] Next, the inspection video acquisition unit 221 of the inspection device 2 sequentially receives the inspection target videos of the work.

[0213] Then, the determination unit 222 of the inspection device 2 operates as follows (1) to (9). (1) The inspection image acquisition means 2221 of the judgment unit 222 acquires a field that meets a condition (the entire frame of the bowl is included) through image recognition processing. Next, the judgment means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "1." Note that "1" is the identifier for the step "Pour soup."

[0214] Next, the judgment means 2222 acquires the current procedure identifier "1" from the current procedure storage unit 214. Next, the judgment means 2222 judges that the procedure identifier "1" acquired by the prediction process matches the current procedure identifier "1", and assigns "normal" to the judgment result information. Next, the judgment means 2222 refers to the procedure information in the storage unit 21, and judges that the next procedure exists. Next, the judgment means 2222 updates the current procedure identifier to the next procedure identifier "2". (2) Next, the inspection image acquisition means 2221 acquires the field that next meets the condition from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a procedure identifier "1".

[0215] Next, the judgment means 2222 acquires the current procedure identifier "2" from the current procedure storage unit 214. Next, since the procedure identifier "1" acquired by the prediction process does not match the current procedure identifier "2", but matches the immediately previous current procedure identifier "1", the judgment means 2222 assigns "normal" to the judgment result information. In other words, when a field indicating the continuation of an operation within one procedure is detected, "normal" is assigned to the judgment result information. Note that after the procedure identifier acquired by the prediction process matches the current procedure identifier and after the current procedure identifier is updated, when the judgment means 2222 acquires a procedure identifier by the prediction process that matches the current procedure identifier immediately before the update, it judges that a normal procedure is continuing, judges that it is normal, and does not judge that it is an error. (3) Typically, the process of (2) is repeated. (4) The inspection image acquisition means 2221 does not acquire fields that do not meet the conditions through image recognition processing (for example, a field in which the edge of the pot overlaps with the worker's hand). Also, fields that do not meet the conditions usually occur consecutively. (5) Next, the inspection image acquisition means 2221 acquires a field that matches the condition from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "2." Note that "2" is the identifier of the step "add noodles."

[0216] Next, the judgment means 2222 acquires the current procedure identifier "2" from the current procedure storage unit 214. Next, the judgment means 2222 judges that the procedure identifier "2" acquired by the prediction process matches the current procedure identifier "2", and assigns "normal" to the judgment result information. Next, the judgment means 2222 refers to the procedure information in the storage unit 21, and judges that the next procedure exists. Next, the judgment means 2222 updates the current procedure identifier to the next procedure identifier "3". (6) Next, the inspection image acquisition means 2221 acquires the next field that meets the conditions from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a procedure identifier "2".

[0217] Next, the determination means 2222 obtains the current procedure identifier "3" from the current procedure storage unit 214. Next, the determination means 2222 assigns "normal" to the determination result information because the procedure identifier "2" obtained by the prediction process does not match the current procedure identifier "3", but matches the immediately preceding current procedure identifier "2". (7) Usually, the process of (6) is repeated. (8) The inspection image acquisition means 2221 does not acquire fields that do not meet the conditions through image recognition processing. (9) Next, the inspection image acquisition means 2221 acquires a field that meets the conditions from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "3." Note that "3" is the identifier of the step "adding ingredients."

[0218] Next, the judgment means 2222 acquires the current procedure identifier "3" from the current procedure storage unit 214. Next, the judgment means 2222 judges that the procedure identifier "3" acquired by the prediction process matches the current procedure identifier "3", and assigns "normal" to the judgment result information. Next, the judgment means 2222 refers to the procedure information in the storage unit 21, and judges that all procedures have been completed.

[0219] Next, the output unit 23 outputs the determination result information.

[0220] (Example 2) Now, worker B is making ramen. First, worker B puts the noodles into the ramen bowl.

[0221] Next, the inspection video acquisition unit 221 of the inspection device 2 sequentially receives the inspection target videos of the work.

[0222] The inspection device 2 operates as follows.

[0223] That is, the inspection image acquisition means 2221 of the judgment unit 222 acquires a field that meets the condition (including the entire frame of the bowl) through image recognition processing. Next, the judgment means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "2." Note that "2" is the identifier for the step "adding noodles."

[0224] Next, the determination means 2222 acquires the current procedure identifier "1" from the current procedure storage unit 214. Next, the determination means 2222 determines that the procedure identifier "2" acquired by the prediction process does not match the current procedure identifier "1", and assigns "error" to the determination result information.

[0225] Next, the output unit 23 outputs the judgment result information. The output unit 23 outputs a buzzer sound as the output of the judgment result information "error", to inform the worker B that the procedure is different. In the case of an error, it is preferable that the output unit 23 outputs a voice or sound corresponding to the judgment result information "error" and informs the worker B that an error has occurred.

[0226] As described above, according to this embodiment, a learning device that can be used by the inspection device can be obtained. Note that the learning device is the first learning device or the second learning device.

[0227] Furthermore, according to the above embodiment, by photographing a task having two or more steps, it is possible to determine in real time whether the task is being performed correctly, and to inform the worker of the result of the determination.

[0228] The processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software for realizing the learning device 3 in this embodiment is a program as follows. That is, this program includes a computer that can access a teacher data storage unit in which one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, is stored, a teacher data acquisition unit that acquires one or more fields that satisfy a predetermined condition from the normal video for each of the two or more steps, and acquires two or more teacher data that are pairs of a procedure identifier that identifies a procedure corresponding to the field and the field, This is a program for functioning as a learning unit that performs machine learning learning processing on the two or more teacher data, acquires a learning device, and accumulates the learning device.

[0229] (Embodiment 3) In this embodiment, an inspection device is described that acquires a video of a series of work steps having two or more steps and equipment-related information, uses the video and the equipment-related information to inspect whether the work is appropriate, and outputs the inspection results.

[0230] In this embodiment, an inspection device that inspects the moving image after inspecting the device-related information will be described.

[0231] In this embodiment, an inspection device that inspects device-related information after inspecting a moving image will be described.

[0232] Furthermore, in this embodiment, an inspection device that uses a machine learning algorithm to inspect whether or not the work is appropriate and outputs the inspection result will be described.

[0233] 14 is a conceptual diagram of an inspection system B in this embodiment. The inspection system B includes one or more cameras 1, an inspection device 4, and one or more devices C. The camera 1 corresponds to the device C. However, the camera 1 and the device C do not have to have a one-to-one correspondence, and may have, for example, a many-to-one or one-to-many correspondence.

[0234] The device C is a device that performs work. The device C is, for example, a robot or a computer. The robot may be an industrial robot, a humanoid robot, or the like, and the type does not matter. The robot is, for example, a so-called drone. The drone is an air vehicle. It is preferable that the drone here is equipped with a camera 1 that takes pictures. It is preferable that the drone here has a function of acquiring and transmitting position information that specifies the flying position. It is preferable that the position information is information that indicates a position in three-dimensional space (for example, (latitude, longitude, altitude)). It is preferable that the drone here has a function of transporting luggage and carrying it to a specified location. In addition, the computer is, for example, a so-called personal computer, a tablet terminal, a smartphone, a server, or the like, and the type does not matter. The device C may be a device whose installation location is fixed, or may be a device (mobile body) that moves. The device C may be integrated with the camera 1.

[0235] 15 is a block diagram of an inspection system B in this embodiment. An inspection device 4 constituting the inspection system B includes a storage unit 41, a processing unit 42, and an output unit 23. The storage unit 41 includes a teacher data storage unit 211, a learning device storage unit 212, a threshold storage unit 213, a normal device related information storage unit 411, and a current procedure storage unit 214. The processing unit 42 includes an inspection video acquisition unit 221, an device related information acquisition unit 421, and a judgment unit 422. The judgment unit 422 includes a first judgment means 4221 and a second judgment means 4222.

[0236] Various types of information are stored in the storage unit 41 of the inspection device 4. The various types of information include, for example, teacher data, a learning device, a threshold value, information related to normal devices (described later), a current procedure identifier, and procedure information.

[0237] Two or more pieces of teacher data are stored in the teacher data storage unit 211. Usually, one or more pieces of teacher data exist for each procedure in the teacher data storage unit 211. Usually, the teacher data for each procedure is associated with a procedure identifier.

[0238] The training data is, for example, information used for comparison with a part or all of the inspection video. The training data is, for example, data based on a normal video (referred to as a positive example as appropriate). The training data includes, for example, one or more fields that constitute the normal video.

[0239] The teacher data (positive example) here is, for example, information based on one or more pieces of normal device-related information. The teacher data here is, for example, a vector having two or more pieces of normal device-related information as elements. The normal device-related information is device-related information when normal work is performed.

[0240] The teacher data (positive example) here may be information having data based on a normal video and data based on one or more pieces of normal device-related information. The teacher data is, for example, a vector having elements of one or more feature amounts acquired from one or more fields constituting a normal video and one or more pieces of normal device-related information. The positive example may include a flag indicating that the task is normal. The positive example may, for example, have a procedure identifier or may be associated with the procedure identifier.

[0241] The normal video may be information in which a simulation device (not shown) simulates normal operations performed by the device C and records the resulting video. The abnormal video may be information in which a simulation device (not shown) simulates abnormal operations that the device may perform and records the resulting video. Note that the simulation device that simulates operations that the device C may perform and outputs the simulation results is a well-known technology, so detailed description will be omitted.

[0242] The device-related information is input device-related information, generator-related information, or output device-related information. The input device-related information is information input to the device C for the performance of the work to be inspected. The generator-related information is information generated within the device C in the performance of the work. The output device-related information is information output from the device C as a result of the performance of the work. The device-related information is, for example, an instruction or data. The input device-related information is, for example, an instruction given to the device C, or an instruction or information input by a user to the device C. The generator-related information is, for example, an event generated by the information input to the device C. When the device C is a moving body such as a drone, the generator-related information is, for example, a signal generated internally by the device C to place the baggage at the scheduled transportation location (for example, a signal to open the arm catching the baggage). The output device-related information is, for example, information transmitted from the device C to another device, a screen or information displayed by the device C, or one or more pieces of location information acquired by the device C (for example, location information identifying the location where the drone flew).

[0243] The one or more teacher data may be information including data based on an abnormal video, which is a video of an abnormal operation, and abnormal device-related information (referred to as a negative example as appropriate). The negative example is, for example, information acquired using an abnormal video and abnormal device-related information. The negative example is, for example, one or more fields constituting the abnormal video and abnormal device-related information. The negative example is, for example, a vector having elements of one or more feature amounts acquired from one or more fields constituting the abnormal video and one or more pieces of abnormal device-related information. The negative example may include, for example, a flag indicating that the operation is abnormal. The negative example may have, for example, a procedure identifier, or may be associated with the procedure identifier. The abnormal device-related information is abnormal information input to the device C, abnormal information occurring within the device C, or abnormal information output from the device C.

[0244] In the learning device storage unit 212, a third learning device is stored in addition to the teacher data storage unit 211 or instead of the teacher data storage unit 211. Furthermore, the learning device storage unit 212 may store the above-mentioned first learning device or the above-mentioned second learning device.

[0245] A third learning device may exist for each procedure in the learning device storage unit 212. That is, each of the two or more third learning devices may be associated with a procedure identifier.

[0246] The third learning device is a learning device used when determining whether the procedure of the work to be inspected is correct. The third learning device is a learning device configured by performing a learning process on two or more pieces of teacher data using a machine learning algorithm. The teacher data here includes, for example, normal device related information or abnormal device related information. In other words, the teacher data that is a positive example includes, for example, data based on a normal video and normal device related information. Also, the teacher data that is a negative example includes, for example, data based on an abnormal video and abnormal device related information.

[0247] When only one third learning device is stored in the learning device storage unit 212, the third learning device is, for example, a learning device that uses each element of a vector obtained from video-based data and one or more pieces of device-related information as explanatory variables and a procedure identifier as a response variable. However, the third learning device may also be a learning device that uses each element of a vector obtained from video-based data as explanatory variables and a procedure identifier as a response variable. Furthermore, the third learning device may also be a learning device that uses each element of a vector obtained from two or more pieces of device-related information as explanatory variables and a procedure identifier as a response variable.

[0248] Furthermore, when two or more third learners paired with procedure identifiers are stored in the learning device storage unit 212, each third learner is, for example, a learner that uses each element of a vector acquired from video-based data and one or more pieces of device-related information as an explanatory variable and a determination result of whether or not the procedure is relevant as an objective variable. Furthermore, such a third learner may be, for example, a learner that uses each element of a vector acquired from video-based data as an explanatory variable and a determination result of whether or not the procedure is relevant as an objective variable. Furthermore, such a third learner may be, for example, a learner that uses each element of a vector acquired from two or more pieces of device-related information as an explanatory variable and a determination result of whether or not the procedure is relevant as an objective variable.

[0249] Moreover, it is preferable that the third learning device is a learning device acquired by a learning device 5 described later.

[0250] As described above, the machine learning algorithm is, for example, deep learning, random forest, decision tree, SVM, etc. However, the machine learning algorithm is not limited. The fact that the machine learning algorithm is not limited applies to both the learning process and the prediction process.

[0251] A threshold value is stored in the threshold value storage unit 213. The threshold value is, for example, a threshold value relating to the similarity between a vector acquired from a field acquired from the inspection video and one or more pieces of device-related information acquired by the device-related information acquisition unit 421, and the teacher data which is a vector. The threshold value is, for example, a threshold value relating to the similarity between a field acquired from the inspection video and a field included in the normal video. The threshold value is, for example, a threshold value relating to the similarity between a vector having elements of one or more pieces of device-related information acquired by the device-related information acquisition unit 421 and a vector having elements of one or more pieces of normal device-related information.

[0252] The threshold value may be stored for each procedure, that is, one or more threshold values ​​may be associated with a procedure identifier.

[0253] The normal device related information storage unit 411 stores one or more pieces of normal device related information. The normal device related information is device related information when normal work is performed. The normal device related information corresponds to a procedure identifier, for example. The normal device related information may be information constituting the teacher data of a positive example. In other words, the normal device related information storage unit 411 may be included in the teacher data storage unit 211.

[0254] The normal device related information storage unit 411 stores, for example, a vector whose elements are two or more pieces of device related information acquired when a normal operation is performed.

[0255] The processing unit 42 performs various types of processing. The various types of processing are, for example, processing performed by the inspection video acquisition unit 221, the device related information acquisition unit 421, and the determination unit 422.

[0256] The inspection video acquisition unit 221 may acquire, for example, an image that is a capture of the screen of the device C. The inspection video acquisition unit 221 may acquire, for example, two or more images that are captures of the screen of the device C. In such a case, the camera 1 is not necessary.

[0257] For example, when the device C is a moving object such as a drone, the inspection video acquisition unit 221 may acquire both an image acquired by an external camera 1 and an image acquired by a camera 1 equipped in the drone, or may acquire only an image acquired by a camera 1 equipped in the drone. The inspection video acquisition unit 221 acquires an image captured by the camera 1, which is, for example, synchronized with the timing at which a signal indicating that the drone has released the cargo it is carrying (placed it at a predetermined location such as the ground or a building) is received, or is one or more images taken close to the timing that satisfies a predetermined condition.

[0258] The device related information acquisition unit 421 acquires one or more pieces of device related information from the device C. The device related information acquisition unit 421 usually acquires device related information from an operating device C. The technology for acquiring device related information from the device C is a publicly known technology, and therefore a detailed description thereof will be omitted.

[0259] The device-related information acquisition unit 421 acquires, for example, one or more pieces of input device-related information input to the device C. The device-related information acquisition unit 421 acquires, for example, one or more pieces of generation device-related information generated within the device C. The device-related information acquisition unit 421 acquires, for example, one or more pieces of output device-related information output by the device C.

[0260] The device related information acquisition unit 421 may receive the device related information from the device C, or may receive the device related information from a device (not shown) that has received the information transmitted from the device C. The method by which the device related information acquisition unit 421 acquires the device related information is not important.

[0261] It is preferable that the device related information acquisition unit 421 acquires the device related information for each procedure in association with a procedure identifier.

[0262] It is preferable that the device-related information acquisition unit 421 acquires a field used by the determination unit 422 for determination and device-related information at a time corresponding to the time of the field. In other words, it is preferable that the field used by the determination unit 422 for determination and the device-related information are synchronized.

[0263] The judgment unit 422 judges whether the procedure of the work to be inspected is correct or not by using the teacher data based on the normal video and the normal device related information. The judgment unit 422 judges whether the procedure of the work to be inspected is correct or not for the inspection video and the device related information by using the teacher data based on the normal device related information and the normal video.

[0264] It is preferable that the judgment unit 422 performs a machine learning prediction process, for example, using a second learning device to determine whether or not each field in the inspection video should be used as a field for determining whether or not the procedure is correct, obtains one or more fields corresponding to the prediction result that the field will be used, and performs the following judgment process using the one or more fields.

[0265] When the device C is a moving body such as a drone, the determination unit 422 determines whether or not the transport of the luggage has been performed normally, for example, using one or more images acquired by the inspection video acquisition unit 221 and a positive example image. The determination unit 422 determines that the transport of the luggage has been performed normally, for example, when an image having a similarity to the positive example image equal to or greater than a threshold is included in one or more images acquired by the inspection video acquisition unit 221. In addition, the determination unit 422 performs a learning process by a machine learning algorithm, for example, using each of the one or more images acquired by the inspection video acquisition unit 221 and a learning device, and obtains a prediction result indicating whether each image is normal or not.

[0266] When the device C is a moving object such as a drone, the determination unit 422 may determine whether the device C has normally held the luggage by using, for example, one or more images synchronized with a signal for holding the luggage and acquired by the inspection video acquisition unit 221 and a positive example image. Such a determination method may use the similarity of the images as described above, or may use a machine learning algorithm.

[0267] When the device C is a moving body such as a drone, the determination unit 422 determines whether the device C moves normally along the planned flight route by using, for example, one or more pieces of position information that are output device related information of the device C and one or more pieces of position information of the positive example. The one or more pieces of position information that are output device related information of the device C are information that specifies the route that has actually flown. The one or more pieces of position information of the positive example are information that specifies the planned flight route. The determination unit 422, for example, calculates a similarity between a vector having one or more pieces of position information that are output device related information and a vector having one or more pieces of position information of the positive example, and determines that the similarity is normal when the similarity is equal to or greater than a threshold value. The determination unit 422, for example, calculates a distance between one or more pieces of position information that are output device related information and a line in a three-dimensional space that specifies the planned flight route, and determines that the distance is normal when the distance is within a threshold value or less than the threshold value. It should be noted that the method by which the determination unit 422 determines whether the flight route is normal is not important.

[0268] The judgment unit 422 judges whether or not the procedure of the work to be inspected is correct, for example, by using any of the following algorithms. The first method is a method in which both the video and the equipment-related information are used at the same time. The second method is a method in which a first inspection is performed using the equipment-related information, and then a second inspection is performed using the video. The third method is a method in which a second inspection is performed using the video, and then a first inspection is performed using the equipment-related information. Each of these methods will be explained below. (1) First Method

[0269] In the first method, there are a case where a third learning device for each procedure is used, a case where one third learning device is used, and a case where vector similarity is used. (1-2) When using the third learning model for each step

[0270] The judgment unit 422 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, and acquires one or more feature amounts of the field. In addition, the judgment unit 422 acquires one or more pieces of device-related information acquired by the device-related information acquisition unit 421 for each procedure. Then, the judgment unit 422 configures a vector having one or more feature amounts and one or more pieces of device-related information as elements for each procedure. Next, the judgment unit 422 acquires a third learning device for each procedure. Next, the judgment unit 422 performs a machine learning prediction process using the vector and the third learning device for each procedure, and acquires a prediction result. Note that the prediction result is information indicating that the work is "normal" or "abnormal". In this case, the third learning device is a binary classification learning device that identifies whether the work is "normal" or "abnormal".

[0271] Moreover, the one or more feature amounts are image feature amounts. The image feature amount is, for example, a spatial feature amount. The spatial feature amount is, for example, a local image feature (SIFT), a pixel value, an activity, a spatiotemporal correlation, a motion vector, and a frequency distribution. Moreover, the activity is, for example, a maximum value and a minimum value of a plurality of pixels, a dynamic range (DR), a difference value between a plurality of pixels, a pixel value distribution of the entire image, a motion vector distribution of the entire image, and one or more spatiotemporal correlations of the entire image.

[0272] It is preferable that the determination unit 422 obtains the prediction result in the procedure currently being performed in real time. (1-2) When using one third learning module

[0273] The judgment unit 422 acquires, for each procedure, one field of the inspection video acquired by the inspection video acquisition unit 221, and uses the one field, the equipment-related information acquired by the equipment-related information acquisition unit 421, and a third learning device to acquire a procedure identifier that identifies the procedure corresponding to the one field acquired by the inspection image acquisition means through a machine learning prediction process, and judges whether the procedure identifier matches the current procedure identifier.

[0274] For example, the determination unit 422 acquires one or more feature amounts from one field to be inspected, and configures a vector having elements of the one or more feature amounts and one or more pieces of device-related information acquired by the device-related information acquisition unit 421. Next, the determination unit 422 performs a machine learning prediction process using the vector and one third learner, acquires a procedure identifier, and determines whether the procedure identifier matches the current procedure identifier. In this case, it is preferable that the third learner is a multi-value classification learner that determines one procedure identifier from multiple procedure identifiers. (1-3) When using vector similarity

[0275] The determination unit 422 obtains, for each procedure, one field of the inspection video obtained by the inspection video obtaining unit 221, and obtains one or more feature amounts of the field. The determination unit 422 also obtains, for each procedure, one or more pieces of device-related information obtained by the device-related information obtaining unit 421. Then, the determination unit 422 configures, for each procedure, a vector whose elements are each of the one or more feature amounts and each of the one or more pieces of device-related information.

[0276] Next, the judgment unit 422 acquires, for each procedure, a vector that is teacher data corresponding to a normal operation from the storage unit 41. Next, the judgment unit 422 calculates the similarity between the constructed vector and the teacher data vector for each procedure. Next, the judgment unit 422 judges whether the similarity is equal to or greater than a threshold value or is larger than a threshold value (whether or not it is normal). (2) Second Method

[0277] The first judgment means 4221 judges whether or not the procedure of the work to be inspected is correct using the equipment-related information and normal equipment-related information acquired by the equipment-related information acquisition unit 421 (referred to as the "first judgment"). Next, when the judgment result of the first judgment means 4221 is that the procedure is correct, the second judgment means 4222 judges whether or not the procedure of the work to be inspected for the inspection video is correct using teacher data based on the normal video (referred to as the "second judgment").

[0278] The following three examples are examples of the first judgment. (2-1-1) When using the third learning model for each step

[0279] In the first judgment, for example, the first judgment means 4221 configures a vector for each procedure, the elements of which are one or more pieces of device-related information acquired by the device-related information acquisition unit 421, and performs a machine learning prediction process using the vector and a third learner corresponding to each procedure to acquire a prediction result indicating whether the operation of the corresponding procedure is normal or not. In this case, there are two or more third learners corresponding to each procedure identifier. In addition, the third learner is a learner that uses one or more pieces of device-related information as explanatory variables and information indicating whether the operation of the corresponding procedure is normal or not as a target variable. (2-1-2) When using one third learning device

[0280] In the first judgment, for example, the first judgment means 4221 uses a machine learning algorithm to construct a vector whose elements are one or more pieces of device-related information acquired by the device-related information acquisition unit 421 for each procedure, and performs machine learning prediction processing using the vector and one third learning device to acquire a procedure identifier. Then, the first judgment means 4221 judges whether the acquired procedure identifier matches the current procedure identifier. If they match, it is normal, and if they do not match, it is abnormal. (2-1-3) When using vector similarity

[0281] In the first judgment, for example, the first judgment means 4221 configures, for each procedure, a vector whose elements are one or more pieces of apparatus-related information acquired by the apparatus-related information acquisition unit 421. Next, the first judgment means 4221 calculates, for each procedure, a similarity between the vector and a vector corresponding to the procedure identifier of the corresponding procedure (a vector corresponding to a normal operation). Next, the first judgment means 4221 judges, for each procedure, whether the similarity is equal to or greater than a threshold value or not (whether it is normal or not).

[0282] The following three are specific examples of the second judgment: (2-2-1) When using the third learning model for each step

[0283] In the second judgment, for example, the second judgment means 4222 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 acquires a third learning device for each procedure. Next, the second judgment means 4222 performs a machine learning prediction process using the vector and the third learning device for each procedure, and acquires a prediction result. Note that the prediction result is information indicating that the work is "normal" or "abnormal". In this case, the third learning device is a binary classification learning device that distinguishes whether the work is "normal" or "abnormal". Also, the third learning device is a learning device that uses two or more feature amounts of the field as explanatory variables. (2-2-2) When using one third learning device

[0284] In the second judgment, for example, the second judgment means 4222 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 acquires one third learning device. Next, the second judgment means 4222 performs a machine learning prediction process using the vector and the third learning device for each procedure, and acquires a procedure identifier. Then, the second judgment means 4222 judges whether the acquired procedure identifier matches the current procedure identifier. (2-2-3) When using vector similarity

[0285] In the second judgment, for example, the second judgment means 4222 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 calculates the similarity between the vector and a vector corresponding to the procedure identifier of the corresponding procedure (a vector corresponding to a normal operation) for each procedure. Next, the second judgment means 4222 judges whether the similarity is equal to or greater than a threshold value (whether it is normal or not) for each procedure. (3) The third method

[0286] The second judgment means 4222 uses teacher data based on the normal video to judge whether or not the procedure of the work to be inspected for the inspection video is correct (performs a "second judgment"). Next, when the judgment result of the second judgment means 4222 is that the procedure is correct, the first judgment means 4221 uses the equipment-related information and normal equipment-related information acquired by the equipment-related information acquisition unit 421 to judge whether or not the procedure of the work to be inspected is correct (performs a "first judgment"). Note that specific processing examples of the second judgment and the first judgment have been described above, so explanations thereof will be omitted here.

[0287] Next, an example of the operation of the inspection system B will be described. First, an example of the operation of the inspection device 4 will be described with reference to the flowchart of Fig. 16. In the flowchart of Fig. 16, the description of the same steps as those in the flowchart of Fig. 3 will be omitted.

[0288] (Step S1601) The device-related information acquisition unit 421 acquires one or more pieces of device-related information corresponding to the i-th procedure. The device-related information corresponding to the i-th procedure is, for example, device-related information acquired from device C at the time or at the time when the one or more fields acquired in step S304 were shot. It is preferable that the acquired device-related information is information synchronized with the one or more fields acquired in step S304.

[0289] (Step S1602) The judgment unit 422 judges whether the work in the i-th procedure is normal or not by using one or more fields acquired in step S304 and one or more pieces of device-related information acquired in step S1601. The process proceeds to step S306. An example of such a judgment process will be described with reference to the flowcharts of FIG. 17 to FIG. 22.

[0290] In the flowchart of FIG. 16, the process ends when the power is turned off or an interrupt occurs to end the process.

[0291] Furthermore, in the flowchart of FIG. 16, the current procedure identifier in current procedure storage unit 214 is updated to the next procedure identifier in accordance with the increment of counter i.

[0292] Next, a first example of the determination process in step S1602 will be described with reference to the flowchart in Fig. 17. In the flowchart in Fig. 17, explanations of steps equivalent to those in the flowchart in Fig. 5 will be omitted.

[0293] (Step S1701) The determination unit 422 acquires a third learning device from the learning device storage unit 212.

[0294] (Step S1702) The determination unit 422 acquires one or more feature amounts from one or more fields acquired in step S304. The feature amount is an image feature amount. The feature amount may be a temporal feature amount in two or more fields. The temporal feature amount is, for example, a motion vector distribution of the entire image, or a spatiotemporal correlation of two or more images as a whole.

[0295] (Step S1703) The determination unit 422 configures a vector having as elements one or more of the feature amounts acquired in step S1702 and one or more of the device-related information acquired in step S1601.

[0296] (Step S1704) The determination unit 422 provides the third learning device acquired in step S1701 and the vector acquired in step S1703 to a module that performs machine learning prediction processing, executes the module, and acquires a procedure identifier. The process proceeds to step S505.

[0297] Next, a second example of the determination process in step S1602 will be described with reference to the flowchart in Fig. 18. In the flowchart in Fig. 18, explanations of steps equivalent to those in the flowcharts in Fig. 5 and Fig. 6 will be omitted.

[0298] (Step S1801) The determination unit 422 determines whether or not a third learning device paired with the i-th procedure identifier exists in the learning device storage unit 212.

[0299] (Step S1802) The determination unit 422 acquires the third learning device paired with the i-th procedure identifier from the learning device storage unit 212. The process proceeds to step S1702.

[0300] (Step S1803) The determination unit 422 provides the third learning device acquired in step S1802 and the vector acquired in step S1703 to a module that performs machine learning prediction processing, executes the module, and acquires a procedure identifier. Then, the process proceeds to step 607.

[0301] Next, a third example of the determination process in step S1602 will be described with reference to the flowchart in Fig. 19. In the flowchart in Fig. 19, explanations of steps equivalent to those in the flowcharts in Fig. 5 and Fig. 17 will be omitted.

[0302] Next, a fourth example of the determination process in step S1602 will be described with reference to the flowchart in Fig. 20. In the flowchart in Fig. 20, explanations of steps equivalent to those in the flowcharts in Fig. 7 and Fig. 17 will be omitted.

[0303] (Step S2001) Determination unit 422 acquires training data corresponding to the i-th procedure from training data storage unit 211. The training data here is a vector constructed using one or more fields capturing images of normal work in the i-th procedure and one or more pieces of device-related information during normal work in the i-th procedure.

[0304] (Step S2002) The judgment unit 422 calculates the similarity between the vector that is the teacher data acquired in step S2001 and the vector constructed in step S1703. Then, the process proceeds to step S706.

[0305] Next, a fifth example of the determination process in step S1602 will be described with reference to the flowchart in Fig. 21. In the flowchart in Fig. 21, explanations of steps equivalent to those in the flowchart in Fig. 5 will be omitted.

[0306] (Step S2101) The first determination means 4221 performs a first determination using one or more pieces of device-related information acquired in step S1601. An example of the first determination process will be described with reference to the flowchart of FIG.

[0307] (Step S2102) If the result of the first judgment in step S2101 is "normal", the first judgment means 4221 proceeds to step S2103, and if the result is "error", the first judgment means 4221 proceeds to step S507.

[0308] (Step S2103) The second determination means 4222 performs a second determination using one or more fields acquired in step S304. Examples of the second determination process are the determination processes in Figs. 5, 6, 7, and 8 in the first embodiment.

[0309] (Step S2104) If the result of the second judgment in step S2103 is "normal", the second judgment means 4222 proceeds to step S506, and if the result is "error", the second judgment means 4222 proceeds to step S507.

[0310] Next, an example of the first determination process in step S2101 will be described with reference to the flowchart in Fig. 22. In the flowchart in Fig. 22, explanations of steps equivalent to those in the flowchart in Fig. 7 will be omitted.

[0311] (Step S2201) The first judgment means 4221 acquires teacher data paired with a corresponding procedure identifier from the teacher data storage unit 211. Note that the teacher data here is a vector configured using two or more pieces of device-related information acquired when a normal operation is performed in the corresponding procedure.

[0312] (Step S2202) The first judgment means 4221 configures a vector using two or more pieces of device related information acquired in step S1601.

[0313] (Step S2203) The first judgment means 4221 calculates the similarity between the training data acquired in step S2201 and the vector acquired in step S2202. Then, the process proceeds to step S706.

[0314] As described above, according to this embodiment, it is possible to determine whether or not a task using device C is being performed correctly by using a video obtained by filming a task having two or more steps performed using device C and device-related information when the task is performed using device C.

[0315] Furthermore, according to this embodiment, after making a judgment using the device-related information, a judgment is made using the inspection image only if the judgment is normal, so that the normality of the work can be judged accurately and efficiently.

[0316] Furthermore, according to this embodiment, after making a judgment using the inspection image, a judgment is made using the device-related information only if the judgment is normal, so that the normality of the work can be judged accurately and efficiently.

[0317] Furthermore, according to this embodiment, by using the device-related information when a task is performed using device C, it is possible to determine, by a machine learning algorithm, whether or not the task is being performed correctly.

[0318] Furthermore, the processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software for realizing the inspection device 4 in this embodiment is a program as follows. That is, this program causes a computer to function as an inspection video acquisition unit that acquires an inspection video, which is a video captured of an inspection target work, which is an inspection target work having two or more procedures, being performed; an equipment-related information acquisition unit that acquires equipment-related information, which is information input to the equipment for the inspection target work, information generated within the equipment, or information output from the equipment; a judgment unit that judges whether the procedure of the inspection target work is correct for the inspection video and the equipment-related information using teacher data based on normal equipment-related information, which is equipment-related information when the normal work is performed, and the normal video; and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

[0319] (Embodiment 4) In this embodiment, a description will be given of a learning device 5. In this embodiment, the learning device 5 constitutes a learning unit used by the inspection device 4 described in the fourth embodiment.

[0320] 23 is a block diagram of a learning device 5 in this embodiment. The learning device 5 includes a learning storage unit 51 and a learning processing unit 52. The learning storage unit 51 includes a teacher data storage unit 511. The learning processing unit 52 includes a teacher data acquisition unit 521 and a learning unit 522.

[0321] Various types of information are stored in the learning storage unit 51. The various types of information are, for example, one or more normal videos, teacher data, and a third learning device.

[0322] Two or more pieces of teacher data for configuring a third learning device are stored in the teacher data storage unit 511. Such teacher data is, for example, a field selected by the user from a normal video, and is a vector (vector of positive examples) whose elements are one or more feature amounts and one or more pieces of normal device related information acquired from the field used in the judgment process of the judgment means 2222.

[0323] The teacher data may be, for example, one or more fields selected by the user from a normal video. The teacher data may be, for example, one or more pieces of normal device related information acquired when the device C operates normally.

[0324] In addition, the teacher data is, for example, one or more fields selected by the user from a video (for example, a normal video or an abnormal video taken when abnormal work is performed), and is a vector (vector of negative examples) whose elements are one or more feature amounts obtained from one or more fields not used in the judgment process of the judgment means 2222 and one or more pieces of abnormal device-related information.

[0325] The teacher data is, for example, one or more fields selected by the user from a video (for example, a normal video or an abnormal video taken when an abnormal task is performed). Also, the teacher data is teaching abnormal device related information.

[0326] The teacher data in the teacher data storage unit 511 may be stored by the learning processing unit 52 or by a means not shown. The method of storing the teacher data in the teacher data storage unit 511 is not important.

[0327] The learning processing unit 52 performs various processes. The various processes are, for example, processes performed by the teacher data acquisition unit 521 and the learning unit 522.

[0328] The teacher data acquisition unit 521 acquires, for each of the two or more procedures, two or more pieces of teacher data, which are combinations of one or more fields, normal device related information, and procedure identifiers, that satisfy a predetermined condition, from the normal video.

[0329] The teacher data acquisition unit 521 acquires, for example, one or more fields that satisfy a predetermined condition and one or more pieces of normal device related information from the normal video for each of two or more steps. Then, the teacher data acquisition unit 521 acquires one or more feature amounts using one or more fields. Next, the teacher data acquisition unit 521 configures teacher data (positive example) that is a vector whose elements are one or more feature amounts and one or more pieces of normal device related information. Next, the teacher data acquisition unit 521 associates the vector with the step identifier of the corresponding step and accumulates it in the teacher data storage unit 511.

[0330] Furthermore, the teacher data acquisition unit 521 acquires, for example, one or more fields that satisfy a predetermined condition and one or more pieces of abnormal device-related information from the abnormal video for each of two or more steps. Next, the teacher data acquisition unit 521 acquires one or more feature amounts from the one or more fields. Next, the teacher data acquisition unit 521 acquires a negative example vector having one or more feature amounts and one or more pieces of abnormal device-related information as elements, and accumulates the vector in the teacher data storage unit 511. The teacher data acquisition unit 521 reads out, for example, two or more pieces of teacher data from the teacher data storage unit 511.

[0331] The process of acquiring the vector may be performed by the learning unit 522.

[0332] The learning unit 522 performs a machine learning learning process using two or more pieces of teacher data acquired by the teacher data acquisition unit 521, acquires a third learning device, and accumulates the third learning device. Note that, as described above, any machine learning algorithm may be used.

[0333] Next, a first operation example of the learning device 5 will be described with reference to the flowchart of Fig. 24. In the flowchart of Fig. 24, the description of the same steps as those in Fig. 10 will be omitted.

[0334] (Step S2401) The teacher data acquiring unit 321 acquires one or more pieces of device related information. The method for acquiring one or more pieces of device related information may be the same as that used by the device related information acquiring unit 421.

[0335] (Step S2402) The teacher data acquisition unit 321 acquires one or more feature amounts from one or more fields acquired in step S1005. Then, the teacher data acquisition unit 321 constructs a vector whose elements are the one or more feature amounts and one or more pieces of device-related information acquired in step S2401, associates the vector with the j-th procedure identifier, and accumulates it in the teacher data storage unit 511. Proceed to step S1007. The vector is teacher data.

[0336] (Step S2403) The learning unit 522 provides the two or more pieces of teacher data stored in the teacher data storage unit 511 in step S2402 to a machine learning learning module, executes the module, and acquires a third learning device. The third learning device here is a learning device that uses each element of the vector as an explanatory variable and the procedure identifier as an objective variable.

[0337] (Step S2404) The learning unit 522 stores the third learning device acquired in step S2403 in the learning storage unit 51. The process ends.

[0338] Next, a second operation example of the learning device 5 will be described with reference to the flowchart in Fig. 25. In the flowchart in Fig. 24, explanations of steps similar to those in Figs. 10, 11, and 24 will be omitted.

[0339] (Step S2501) The learning unit 522 provides one or more positive examples and one or more negative examples to a module that performs a machine learning learning process, executes the module, and acquires a third learning device.

[0340] (Step S2502) The learning unit 522 accumulates the third learning device acquired in step S2501 in a pair with the procedure identifier of the k-th procedure. Note that the third learning device here uses each element of the vector as an explanatory variable and information indicating whether or not the procedure is a corresponding procedure as a target variable. The third learning device here is a learning device for performing binary classification.

[0341] As described above, according to this embodiment, a learning device that can be used by the inspection device 4 is obtained. The learning device is a third learning device. The inspection device 4 may also acquire the first learning device or the second learning device acquired by the inspection device 2. More specifically, according to this embodiment, by capturing an image of an operation having two or more steps and acquiring device-related information from the device C, a learning device for determining whether the operation is being performed correctly in real time can be acquired.

[0342] In addition, the program for realizing the learning device 5 in this embodiment is a program for causing a computer that can access a teacher data storage unit in which one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, being performed and normal equipment-related information, which is equipment-related information when the normal operation is performed, to function as a teacher data acquisition unit that acquires, for each of the two or more steps, from the normal video, two or more partial teacher data that are pairs of one or more fields that satisfy predetermined conditions, the normal equipment-related information, and a procedure identifier, and a learning unit that performs machine learning learning processing on the two or more partial teacher data, acquires a learning device, and accumulates the learning device.

[0343] FIG. 26 shows the appearance of a computer that executes the program described in this specification to realize the inspection device 2 or learning device 3 of the various embodiments described above. The above-mentioned embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 26 is an overview of this computer system 300, and FIG. 27 is a block diagram of the system 300.

[0344] In FIG. 26, a computer system 300 includes a computer 301 including a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

[0345] 27, computer 301 includes, in addition to CD-ROM drive 3012, MPU 3013, bus 3014 connected to CD-ROM drive 3012 etc., ROM 3015 for storing programs such as a boot-up program, RAM 3016 connected to MPU 3013 for temporarily storing instructions of application programs and providing temporary storage space, and hard disk 3017 for storing application programs, system programs, and data. Although not shown here, computer 301 may further include a network card for providing connection to a LAN.

[0346] A program for causing computer system 300 to execute functions of inspection device 2 and the like of the above-mentioned embodiment may be stored on CD-ROM 3101, inserted into CD-ROM drive 3012, and further transferred to hard disk 3017. Alternatively, the program may be transmitted to computer 301 via a network (not shown) and stored on hard disk 3017. The program is loaded into RAM 3016 when executed. The program may be loaded directly from CD-ROM 3101 or the network.

[0347] The program does not necessarily include an operating system (OS) or a third party program that causes the computer 301 to execute the functions of the inspection device 2 of the above-mentioned embodiment. The program only needs to include an instruction portion that calls appropriate functions (modules) in a controlled manner to obtain a desired result. How the computer system 300 operates is well known, and a detailed description will be omitted.

[0348] In addition, in the above program, the steps of transmitting information and receiving information do not include processing performed by hardware, such as processing performed by a modem or interface card in the transmitting step (processing that is performed only by hardware).

[0349] Furthermore, the computer that executes the above program may be a single computer or a plurality of computers. That is, centralized processing or distributed processing may be performed. That is, the inspection device 2, etc. may be a standalone device or may be composed of two or more devices.

[0350] Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be physically realized by one medium.

[0351] In each of the above embodiments, each process may be realized by centralized processing in a single device, or may be realized by distributed processing in a plurality of devices. [Industrial Applicability]

[0352] INDUSTRIAL APPLICABILITY As described above, the inspection device according to the present invention has the effect of being able to determine whether or not an operation having two or more steps is being performed correctly by photographing the operation, and is useful as an inspection device, etc. [Explanation of symbols]

[0353] A, B Inspection System C equipment 1 Camera 2, 4 Inspection equipment 3.5 Learning Device 21, 41 Storage area 22, 42 Processing section 23 Output section 31, 51 Learning storage section 32, 52 Learning processing unit 211, 511 Teacher data storage unit 212 Learning Unit Storage Unit 213 Threshold storage unit 214 Current procedure storage section 221 Inspection video acquisition unit 222, 422 Judgment Department 321, 521 Teacher Data Acquisition Department 322, 522 Learning Department 411 Normal equipment related information storage section 421 Equipment Related Information Acquisition Department 2221 Inspection image acquisition means 2222 Judgment means 4221 First means of judgment 4222 Second judgment means

Claims

1. A current procedure storage unit in which a current procedure identifier that identifies a procedure currently being inspected among two or more procedures is stored; an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more procedures, is being performed; an apparatus-related information acquisition unit that acquires apparatus-related information, which is information input to the apparatus for the work to be inspected, information generated within the apparatus, or information output from the apparatus; a learning module storage unit in which a learning module configured by performing a learning process using a machine learning algorithm on two or more pieces of teacher data, each of which is a set of one field corresponding to each of two or more procedures, normal device-related information that is device-related information when normal work is performed, and a procedure identifier that identifies each procedure, is stored; a determination unit that obtains one field of the inspection video obtained by the inspection video obtaining unit, obtains a procedure identifier that identifies a procedure corresponding to the one field by a machine learning prediction process using the one field, the device-related information obtained by the device-related information obtaining unit, and the learning device, and determines whether the procedure identifier matches the current procedure identifier; and an output section that outputs judgment result information relating to the judgment result of the judgment section.

2. A storage unit in which procedure information for identifying two or more procedures constituting a task is stored; a current procedure storage unit in which a current procedure identifier that identifies a procedure currently being inspected among the two or more procedures is stored; an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more procedures, is being performed; A judgment unit that judges whether or not the procedure of the inspection target work for the inspection video is correct using teacher data based on a normal video, which is a video captured of a normal work being performed, which is a correct work having two or more procedures; an output unit that outputs judgment result information regarding the judgment result of the judgment unit, The teacher data is having one or more fields corresponding to each of the two or more procedures, The determination unit is an inspection image acquisition means for acquiring one or more fields of the inspection video acquired by the inspection video acquisition unit; a determination means for determining whether or not a procedure corresponding to the one or more fields acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier, using the one or more fields acquired by the inspection image acquisition means and one or more fields corresponding to the current procedure identifier; The determination means is An inspection device that, when the procedure corresponding to the one or more fields acquired by the inspection image acquisition means matches the procedure identified by the current procedure identifier, overwrites the current procedure identifier with the procedure identifier of the next procedure indicated by the procedure information.

3. The inspection video acquisition unit includes: The inspection device according to claim 2 , wherein the inspection video is sequentially acquired while an inspection target operation is being performed.

4. The teacher data is A pair of a field corresponding to each of the two or more procedures and a procedure identifier for identifying each procedure, Further comprising a learning module storage unit in which a learning module configured by performing a learning process on two or more pieces of teacher data using a machine learning algorithm is stored; The determination means is 4. The inspection device according to claim 2, further comprising: a machine learning prediction process using the one field acquired by the inspection image acquisition means and the learning device, to acquire a procedure identifier corresponding to the one field, and to determine whether the procedure identifier matches the current procedure identifier.

5. The determination means is An inspection device as described in claim 2 or claim 3, which calculates a similarity between a field corresponding to each of the two or more procedures and the one field acquired by the inspection image acquisition means, and uses the two or more similarities to determine whether the procedure corresponding to the one field acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

6. The determination unit is An inspection device as described in any one of claims 2 to 5, which obtains one or more fields that are a portion of the fields from the inspection video that satisfy predetermined conditions, and uses the one or more fields and the teacher data to determine whether the procedure of the work to be inspected for the inspection video is correct.

7. The determination unit is An inspection device as described in any one of claims 1 to 6, which performs a machine learning prediction process using a second learning device to determine whether or not each field in the inspection video will be used as a field for determining whether or not it is a correct procedure, obtains one or more fields corresponding to the prediction result that they will be used, and uses the one or more fields and the teacher data to determine whether or not the procedure of the work to be inspected for the inspection video is correct.

8. a teacher data storage unit in which two or more teacher data are stored based on a normal video, which is a video of a normal operation, which is a correct operation having two or more steps, being performed on a device, the normal video having two or more fields corresponding to a step identifier that identifies each of the two or more steps, and normal device-related information, which is device-related information of the device on which the normal operation was performed; a teacher data acquisition unit that acquires, for each of the two or more procedures, from the normal video, two or more teacher data that are a combination of an explanatory variable having one or more fields that satisfy a predetermined condition and the normal device related information, and a target variable that is a procedure identifier corresponding to the one or more fields; A learning device comprising: a learning unit that performs machine learning learning processing using the two or more teacher data acquired by the teacher data acquisition unit, acquires a learning device, and accumulates the learning device.

9. An inspection method comprising all of the processes performed by the inspection device described in any one of claims 1 to 7.

10. A learning method comprising all of the processes performed by the learning device described in claim 8.

11. A computer comprising: A program for causing the inspection device according to any one of claims 1 to 7 to function as such an inspection device.

12. A computer comprising: A program for causing the learning device according to claim 8 to function.

Images