Work management system, learning device, inference device, control method and program

Abstract

A training data acquisition unit (410) of a work management system (1) acquires training data including image data captured by a camera (130) in advance, a trained model generation unit (420) generates a trained model by machine learning using the training data, and an inference result data generation unit (240) inputs inference data to the trained model to generate inference result data enabling identification of a work position shown by new image data. The training data includes first image data captured toward a first work position, and second image data captured toward a second work position, and the trained model generation unit (420) uses classification learning to generate a trained model capable of classifying each image data on the basis of data obtained from the periphery of a work position.

Description

[Technical Field]

[0001] The present disclosure relates to a work management system, a learning device, an inference device, a control method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known a work management system that uses a camera to identify and manage work performed by a worker using a tool on a workpiece as a processing object at a work site such as a factory.

[0003] Patent Document 1 discloses a work management system that calculates the three-dimensional position of a marker attached to a tool using image data obtained when the marker is captured by a camera, and compares the calculated position with the three-dimensional position of the tool during a predetermined task to determine whether the task is correct or incorrect.In the system disclosed in Patent Document 1, the camera is installed at a position away from the tool, and when a worker works in a complicated place, there are many blind spots for the camera, so if the marker cannot be captured, the three-dimensional position of the tool cannot be calculated and it is not possible to determine whether the task is correct or incorrect.

[0004] In response to this, Patent Document 2 discloses a tool system that identifies a set work target, which is a work target on which a tool is set, among multiple work targets based on an image captured by an imaging unit provided on the tool. In Patent Document 2, the imaging unit is attached to the tip of a conductor and is configured so that a socket attached to the output shaft of the tool falls within the imaging range. Furthermore, in the system disclosed in Patent Document 2, a pattern recognition process is performed on video-format images captured by the imaging unit when a user works with the tool, using a reference image including multiple frames corresponding to multiple work targets as template data, to identify the set work target. Specifically, in the pattern recognition process, the captured image is input into a trained model created by machine learning using the template data, and the set work target is identified by recognizing objects reflected in the captured image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-132736 [Patent Document 2] Japanese Patent Publication No. 2021-178397 Summary of the Invention [Problem to be solved by the invention]

[0006] In the system disclosed in Patent Document 1, the three-dimensional position of a marker is calculated using image data obtained from a camera to determine whether the task is correct or incorrect, so there is a risk that discrimination will be difficult if, for example, the resolution of the image data is low or the lens is out of focus and the boundary between the image of the marker and other images becomes unclear. Similarly, in the system disclosed in Patent Document 2, the object shown in the captured image is recognized by pattern recognition processing, so there is a risk that recognition of the object shown will be difficult if, for example, the resolution of the captured image is low or the lens is out of focus and the boundary between the image of the object shown and other images becomes unclear.

[0007] The present disclosure has been made in consideration of the above-described circumstances, and aims to improve the recognition rate of work performed on a workpiece using a tool. [Means for solving the problem]

[0008] In order to achieve the above object, the work management system according to the present disclosure is a work management system that manages work on a workpiece as a processing object using a tool, and includes: a learning data acquisition unit that acquires learning data including image data previously captured by an imaging member that captures an image toward a work position where work is being performed on the workpiece using the tool; a trained model generation unit that generates a trained model for inferring the work position from new image data by machine learning using the learning data; and an inference data acquisition unit that acquires the new image data captured by the imaging member as inference data; Every time the inference data acquisition unit acquires inference data,an inference result data generation unit that inputs inference data into a trained model and generates inference result data that can identify a work position; picture, The learning data includes first image data obtained by imaging toward a first work position where a first task is being performed, and second image data obtained by imaging toward a second work position where a second task different from the first task is being performed, and the trained model generation unit generates a trained model that can classify the first image data and the second image data from data obtained from around the work position by classification learning as machine learning. death, The learning data includes original image data obtained by capturing an image in advance using an imaging member, and processed image data obtained by performing image processing on the original image data. The original image data includes at least first original image data obtained by capturing an image using an imaging member from a first angle, and second original image data obtained by capturing an image using an imaging member from a second angle different from the first angle. The processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce changes in the situation in which the imaging member captures an image. The system further includes a work order determination unit that determines whether the order of work tasks on the workpiece is correct by comparing work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with work position order data predetermined based on the order of work tasks on the workpiece, and a tool operation control unit that enables operation of a tool to be used for the next work task on the workpiece when the work order determination unit determines that the order of work tasks on the workpiece is correct, wherein the tools include a first tool and a second tool having a different use from the first tool based on a function realized by an output operation performed by the power unit, and the tool operation control unit enables operation of the other tool when the tool used for the previous work task that the work order determination unit determines to be in the correct order is one of the first tool and the second tool and the tool to be used for the next work task is the other of the first tool and the second tool. . [Effects of the Invention]

[0009] According to the present disclosure, the trained model generation unit generates a trained model capable of classifying first image data and second image data from data obtained from the vicinity of the work position through classification learning, and the inference result data generation unit can input inference data to the trained model to generate inference result data capable of identifying whether the work position is the first work position or the second work position. As a result, the work management system according to the present disclosure has a higher recognition rate for work performed using tools on workpieces than a work management system that does not generate a trained model capable of classifying first image data and second image data through classification learning. [Brief explanation of the drawings]

[0010] [Figure 1] Overall explanatory diagram of a work management system according to a first embodiment [Figure 2] FIG. 1 is a diagram showing a functional configuration of a work management system according to a first embodiment; [Figure 3] FIG. 1 is a block diagram showing the hardware configuration of each device according to the first embodiment. [Figure 4A] FIG. 1 is an explanatory diagram of image data and date and time data according to the first embodiment; [Figure 4B] FIG. 1 is an explanatory diagram of learning data according to the first embodiment; [Figure 5] FIG. 1 is an explanatory diagram of a neural network according to a first embodiment; [Figure 6] FIG. 10 is a diagram showing an example of displaying score data of inference result data according to the first embodiment; [Figure 7] Flowchart of trained model generation processing according to embodiment 1 [Figure 8] Flowchart of inference result data generation processing according to the first embodiment [Figure 9] FIG. 10 is a diagram showing the functional configuration of a work management system according to a second embodiment; [Figure 10] FIG. 10 is a diagram showing the functional configuration of a work management system according to a third embodiment. [Figure 11] FIG. 10 is a diagram showing an example of displaying an out-of-order image according to the third embodiment. [Figure 12] Flowchart of work order determination process according to the third embodiment DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a work management system, a learning device, an inference device, a control method, and a program according to embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are designated by the same reference numerals.

[0012] [Embodiment 1] (Regarding the Work Management System 1 According to the First Embodiment) A work management system 1 according to a first embodiment of the present disclosure is a system for managing work on a workpiece, which is an object to be processed at a manufacturing site such as a factory. As shown in Fig. 1, the work management system 1 includes an intelligent tool 100 as an example of a tool used by a worker, and a data collection device 200 that collects data from the intelligent tool 100. The work management system 1 also includes a personal computer 300 installed with software capable of realizing an engineering environment used by the user, and a learning device 400 that learns about a work position, which is a position where the worker is performing work on the workpiece using the intelligent tool 100.

[0013] The intelligent tool 100 and the data collecting device 200 can transmit and receive data via a wireless LAN (Local Area Network) using a wireless communication standard such as Wi-Fi (registered trademark). The data collecting device 200 and the personal computer 300 can transmit and receive data via a wired LAN. The personal computer 300 and the learning device 400 can transmit and receive data via the Internet.

[0014] (Regarding the Intelligent Tool 100 According to the First Embodiment) The intelligent tool 100 is a tool equipped with the functionality of a computer device capable of performing data processing using a program. The intelligent tool 100 includes multiple types of intelligent tools with different uses, such as a first intelligent tool 101, a second intelligent tool 102, ..., and a ninth intelligent tool 109. The intelligent tools 101 to 109 only differ in the function of the power unit, which will be described later, and are otherwise similar in function. Therefore, in order to reduce redundant explanation, "each of the intelligent tools 101 to 109" will be simply referred to as "the intelligent tool 100" below.

[0015] As shown in FIG. 2 , the intelligent tool 100 includes a power supply unit 110 that supplies power to the tool, a power unit 120 that performs output operations according to the tool's intended use, an imaging unit that captures images of the work position where the power unit 120 performs the output operation, and a camera 130 as an example of an imaging member. The power supply unit 110 is a power supply member such as a lithium-ion battery, and the power unit 120 and the camera 130 are operable with power supplied from the power supply unit 110. In this embodiment, the camera 130 includes, for example, an imaging sensor and an illumination light output device (not shown). The camera 130 may also be provided with various mechanisms, such as an autofocus mechanism and an automatic exposure adjustment mechanism (not shown). The camera 130 also includes the functions of the computer device described above. Specifically, the camera 130 includes an image data acquisition unit 131 that acquires image data by imaging, a data transmission unit 132 that transmits the data, and a tool status monitoring unit 133 that monitors the operating status of the tool.

[0016] (Regarding the data collection device 200 according to the first embodiment) The data collection device 200, which is an example of an inference device, is a computer device serving as a so-called database server that accumulates data collected from the intelligent tool 100. The data collection device 200 includes a data receiving unit 210 that receives data, a data storage unit 220 that stores data, a data transmitting unit 230 that transmits data, and an inference result data generating unit 240 that generates inference result data (described later) using a trained model (described later).

[0017] (Regarding the personal computer 300 according to the first embodiment) The personal computer 300 includes a data receiving unit 310 that receives data, a learning data generating unit 320 that generates learning data described below, a setting data generating unit 330 that generates various setting data for configuring the operation of the work management system 1, and a data transmitting unit 340 that transmits data.

[0018] (Regarding the learning device 400 according to the first embodiment) The learning device 400 is a computer device serving as a cloud server for machine learning, equipped with, for example, a computing device such as a central processing unit (CPU) or a graphics processing unit (GPU) capable of high-speed data processing. The learning device 400 includes a learning data acquisition unit 410 that acquires learning data (described later), a trained model generation unit 420 that generates a trained model (described later), and a trained model output unit 430 that outputs the trained model. Since the generation and output of the trained model only requires the advance acquisition of learning data, the load on the network can be reduced even if the amount of information in the learning data required for machine learning increases. Therefore, by using the learning device 400 as a cloud server, the machine learning function can be integrated into a cloud system, which is suitable for actually building the work management system 1.

[0019] (Hardware Configuration of the Intelligent Tool 100 According to the First Embodiment) The following describes the hardware configuration of the intelligent tool 100. In this embodiment, the functions of the intelligent tool 100 as a computer device are included in the camera 130. Therefore, in the following description, the "intelligent tool 100" can be read as the "camera 130."

[0020] 3, the intelligent tool 100 includes a control unit 51 that executes processing in accordance with a control program 59. The control unit 51 includes a CPU. In accordance with the control program 59, the control unit 51 functions as an image data acquisition unit 131 and a tool state monitoring unit 133 shown in FIG.

[0021] 3, the intelligent tool 100 includes a main memory unit 52 into which the control program 59 is loaded and which is used as a work area for the control unit 51. The main memory unit 52 includes a RAM (Random Access Memory).

[0022] The intelligent tool 100 also includes an external storage unit 53 that stores a control program 59 in advance. The external storage unit 53 supplies data stored by the program to the control unit 51 in accordance with instructions from the control unit 51, and stores the data supplied from the control unit 51. The external storage unit 53 includes a nonvolatile memory such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0023] The intelligent tool 100 also includes an operation unit 54 that is operated by a user. Input information is supplied to the control unit 51 via the operation unit 54. The operation unit 54 includes information input components such as a keyboard, a mouse, and a touch panel.

[0024] The intelligent tool 100 may include a display unit 55 that displays information input via the operation unit 54 and information output by the control unit 51. In this case, the display unit 55 may include a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0025] The intelligent tool 100 also includes a transceiver 56 for transmitting and receiving information. The transceiver 56 includes information communication components such as a communication network terminal connected to a network and a wireless communication device. The transceiver 56 functions as the data transmitter 132 shown in FIG. 2.

[0026] Returning to FIG. 3, in the intelligent tool 100, the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55 and the transmission / reception unit 56 are all connected to the control unit 51 via the internal bus 50.

[0027] 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes an image data acquisition step performed by the image data acquisition unit 131, a data transmission step performed by the data transmission unit 132, and a tool state monitoring step performed by the tool state monitoring unit 133.

[0028] (Hardware Configuration of Data Collection Device 200 According to Embodiment 1) 3, the data collection device 200 includes the above-mentioned control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the inference result data generation unit 240 shown in FIG. 2 in accordance with a control program 59. The external memory unit 53 functions as the data storage unit 220 shown in FIG. 2. The transmission / reception unit 56 functions as the data reception unit 210 and data transmission unit 230 shown in FIG. 2.

[0029] 3, the data collection device 200 realizes the functions of the above-described units 210 to 240 shown in FIG. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the data collection device 200 executes a trained model acquisition step performed by the data receiving unit 210 as an example of a trained model acquisition unit and an inference data acquisition unit, and a data reception step as an example of an inference data acquisition step. Furthermore, for example, the data collection device 200 executes a data storage step performed by the data storage unit 220, a data transmission step as an example of an inference result data output step performed by the data transmission unit 230 as an example of an inference result data output unit, and an inference result data generation step performed by the inference result data generation unit 240.

[0030] (Hardware Configuration of Personal Computer 300 According to Embodiment 1) 3, similar to the data collecting device 200, the personal computer 300 also includes the above-mentioned control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the learning data generation unit 320 and the setting data generation unit 330 shown in FIG. 2 in accordance with a control program 59. The transmission / reception unit 56 functions as the data reception unit 310 and the data transmission unit 340 shown in FIG. 2.

[0031] 3, the personal computer 300 realizes the functions of the above-mentioned units 310 to 340 shown in Fig. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the personal computer 300 executes a data reception step performed by the data reception unit 310, a learning data generation step performed by the learning data generation unit 320, a setting data generation step performed by the setting data generation unit 330, and a data transmission step performed by the data transmission unit 340.

[0032] (Hardware Configuration of Learning Device 400 According to Embodiment 1) 3, like the data collection device 200 and the personal computer 300, the learning device 400 also includes the above-mentioned control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the trained model generation unit 420 shown in FIG. 2 in accordance with a control program 59. The transmission / reception unit 56 functions as the training data acquisition unit 410 and trained model output unit 430 shown in FIG. 2.

[0033] 3, the learning device 400 realizes the functions of the above-mentioned units 410 to 430 shown in Fig. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the learning device 400 executes a learning data acquisition step performed by the learning data acquisition unit 410, a trained model generation step performed by the trained model generation unit 420, and a trained model output step performed by the trained model output unit 430.

[0034] (Details of the Functional Configuration of the Intelligent Tool 100 According to the First Embodiment) The tool state monitoring unit 133 monitors the operating state of the intelligent tool 100 by monitoring the output operation of the power unit 120. For example, if the power unit 120 is configured as an electric screwdriver, the tool state monitoring unit 133 can detect the operating state from the start of screw tightening by the power unit 120 to the completion of screw tightening by monitoring the state of the power unit 120 from the start of operation to the completion of operation. The tool state monitoring unit 133 also acquires tool-related data on the intelligent tool 100 by acquiring data output from the power supply unit 110, the power unit 120, and the camera 130. Here, the tool-related data is, for example, data indicating a torque value when a screw is tightened and data indicating a torque waveform acquired by a sensor (not shown) of the power unit 120.

[0035] 4A, the image data acquisition unit 131 acquires image data obtained by an imaging sensor (not shown) of the camera 130 and date and time data indicating the date and time when the image data was acquired. Note that, when the tool state monitoring unit 133 detects that the operation of the power unit 120 has been completed, it causes the image data acquisition unit 131 to acquire image data. Therefore, the image data acquired by the image data acquisition unit 131 is image data of the work position and its surroundings when the operation of the power unit 120 has been completed.

[0036] The image data acquisition unit 131 acquires image data for each work position where work is performed with the intelligent tool 100. Therefore, the image data includes, for example, first image data obtained by capturing an image toward a first work position where a first work is performed, and second image data obtained by capturing an image toward a second work position where a second work different from the first work is performed. The image data acquisition unit 131 also acquires multiple types of image data captured from different angles for each work position. That is, the image data includes, for example, multiple types of first image data and multiple types of second image data.

[0037] The data transmission unit 132 transmits the image data and date and time data acquired by the image data acquisition unit 131 to the data collection device 200. In addition, the data transmission unit 132 transmits the tool-related data acquired by the tool state monitoring unit 133 to the data collection device 200.

[0038] (Details of Functional Configuration of Data Collection Device 200 According to Embodiment 1) The data receiving unit 210 receives various types of data, such as the image data, date and time data, and tool-related data, from the intelligent tool 100. The data collecting device 200 stores the various types of data received by the data receiving unit 210 in the data storage unit 220.

[0039] The data transmission unit 230 transmits image data stored in the data storage unit 220 to the personal computer 300. For example, the data transmission unit 230 accesses an engineering environment realized by software installed on the personal computer 300. The data transmission unit 230 then transmits the image data to the personal computer 300 by uploading the image data stored in the data storage unit 220 in the accessed engineering environment as data to be used as learning data, which will be described later.

[0040] (Details of the Functional Configuration of Personal Computer 300 According to Embodiment 1) The data receiving unit 310 receives image data from the data collecting device 200, and the learning data generating unit 320 generates learning data including the image data received by the data receiving unit 310. The learning data generating unit 320 generates data as learning data by linking work position identification data that can identify a work position with image data of the work position. The learning data generating unit 320 also performs image processing on the image data, such as changing the brightness, rotating, enlarging, and cropping the image. The learning data generating unit 320 then links the processed image data, together with the original image data, with the work position identification data, to generate learning data.

[0041] For example, as shown in FIG. 4B , the learning data generation unit 320 generates a folder for each work position as work position identification data, and includes image data for the corresponding work position in the folder to generate learning data linking the work position identification data with image data. For example, the learning data generation unit 320 generates learning data including: first learning data linking first work position identification data, which is work position identification data indicating a first work position, with m types of first image data; second learning data linking second work position identification data, which is work position identification data indicating a second work position, with m types of second image data; ..., n-th learning data linking n-th work position identification data, which is work position identification data indicating an n-th work position, with m types of n-th image data. In this case, the m types of first image data include first image data after the above-described image processing, and the m types of second image data also include second image data after the above-described image processing. Furthermore, the m types of n-th image data also include n-th image data after the above-described image processing.

[0042] The learning data generation unit 320 may determine that the image data are of the same work position if the difference between the dates and times at which the image data were captured is equal to or less than a predetermined threshold, for example, one minute, based on the date and time data for each acquired image data. Classifying the image data by the date and time at which the image data were captured in this manner simplifies the process for the learning data generation unit 320 to link the work position identification data with the image data, making it easier to generate learning data. The learning data generation unit 320 may also classify the image data based on, for example, the similarity of the images. In this case, the data collection device 200 may generate a new trained model capable of calculating the similarity of images by machine learning such as deep learning, or may separately acquire the new trained model from the learning device 400. In this case, the data collection device 200 may need to be able to calculate the similarity of images by itself, for example, by a method such as feature point extraction, without acquiring a new trained model.

[0043] The setting data generation unit 330 generates setting data used by the data collection device 200. The setting data includes data that enables the data collection device 200 to identify data to be linked to inference result data (described later) and transmitted to the external device 600 from among various data stored in the data storage unit 220. Here, the external device 600 is, for example, a relational database that stores data transmitted by the data collection device 200 for data analysis. The setting data also includes data that identifies the external device 600. The setting data also includes data indicating a format, such as a file format or message format, of data to be used when transmitting data to the external device 600, such as extensible markup language (XML) or JavaScript object notification (JSON). The setting data also includes data indicating a communication protocol, such as transmission control protocol (TCP) or message queueing telemetry transport (MQTT), to be used when transmitting data to the external device 600.

[0044] The data transmission unit 340 transmits the learning data to the learning device 400. The data transmission unit 340 also transmits the setting data to the data collection device 200.

[0045] (Details of the Functional Configuration of Learning Device 400 According to Embodiment 1) The learning data acquisition unit 410 acquires learning data from the personal computer 300, and the trained model generation unit 420 generates a trained model for inferring the work position from new image data by machine learning using the learning data acquired by the learning data acquisition unit 410.

[0046] The trained model generation unit 420 learns the relationship between image data and work positions based on training data created based on a combination of image data and work position identification data linked to the image data. That is, it generates a trained model that infers the most likely work position from new image data.

[0047] The learning algorithm used by the trained model generation unit 420 can be any known algorithm, such as supervised learning, unsupervised learning, or reinforcement learning. In this embodiment, a case where a neural network is applied will be described as an example of the learning algorithm. The trained model generation unit 420 learns the relationship between image data and work positions, for example, by so-called supervised learning in accordance with a neural network model. Here, supervised learning refers to a technique in which a learning device is provided with pairs of input and label data, which are results, to learn the features of the learning data and infer a result from the input.

[0048] A neural network consists of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0049] For example, in a three-layer neural network like the one shown in Figure 5, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the middle layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result changes depending on the values ​​of weights W1 and W2.

[0050] In this embodiment, the neural network learns the relationship between image data and work positions through so-called supervised learning in accordance with learning data created based on a combination of image data acquired by the learning data acquisition unit 410 and work position identification data to which the image data is linked.

[0051] That is, the neural network learns by inputting image data to the input layer and adjusting the weights W1 and W2 so that the result output from the output layer approaches a work position that can be identified by the work position identification data.

[0052] The trained model generation unit 420 generates and outputs a trained model by performing the above-described learning. In this way, the trained model generation unit 420 can generate a trained model that can classify the first image data and the second image data included in the training data using a neural network, which is an example of classification learning as machine learning. Furthermore, the trained model generation unit 420 can set the number of types of work position identification data included in the training data to a predetermined number of categories and perform classification learning for each image data included in the training data.

[0053] The trained model output unit 430 outputs the trained model by transmitting the trained model to the personal computer 300. In the personal computer 300, the data receiving unit 310 receives the trained model from the learning device 400, and the data transmitting unit 340 transmits the trained model to the data collecting device 200.

[0054] (Details of the functional configuration of the data collection device 200 as an inference device) The data receiving unit 210, which is an example of a trained model acquisition unit, acquires the trained model by receiving the trained model from the personal computer 300. The data collecting device 200 stores the trained model in the data storage unit 220, which is an example of a trained model storage unit. The data receiving unit 210, which is an example of an inference data acquisition unit, receives new image data from the intelligent tool 100 and acquires the new image data as inference data.

[0055] In the data collection device 200, the inference result data generation unit 240 inputs the inference data into the trained model to generate inference result data capable of identifying a work position, and stores the generated inference result data in the data storage unit 220. Here, the inference result data is data including score data indicating the score value for each type of work position calculated by the trained model. Therefore, the inference result data is data indicating that the work position with the maximum score value is most likely to be the work position indicated by the new image data. Note that the inference result data is data indicating that the work position indicated by the new image data cannot be identified if the maximum score value does not exceed a predetermined threshold.

[0056] 6, the inference result data includes score data indicating that the first work position has a score of "91," the second work position has a score of "2," ..., and the nth work position has a score of "1." In this case, if the predetermined threshold is "40," for example, the inference result data indicates that the first work position with the maximum score value is most likely to be the work position indicated by the new image data.

[0057] Furthermore, in the data collection device 200, the data receiving unit 210 receives setting data from the personal computer 300, and also receives data necessary for managing production from a production management device 500 that is provided outside the work management system 1 and manages the production of workpieces. Here, the data necessary for managing production includes, for example, worker identification data that can identify the worker, such as the worker's employee number, and workpiece identification data that can identify the workpiece, such as the workpiece's serial number. The data collection device 200 also stores these various types of data received by the data receiving unit 210 in the data storage unit 220.

[0058] Furthermore, based on the setting data, the data collection device 200 identifies data to be linked to the inference result data and transmitted to the external device 600 from various types of data such as worker identification data, inference data, date and time data, tool-related data, and workpiece identification data. Then, the data transmission unit 230 transmits the inference result data and the identified data in a linked state.

[0059] (Regarding Trained Model Generation Process According to Embodiment 1) Next, the operation of the learning device 400 to generate and output a trained model will be described using a flowchart. When the learning device 400 is powered on, it starts executing the trained model generation process shown in FIG. 7. First, the learning data acquisition unit 410 acquires learning data from the personal computer 300 (step S101). For example, the learning data acquisition unit 410 acquires, as the learning data, first learning data in which first work position identification data, which is work position identification data indicating the first work position shown in FIG. 4B, is linked to multiple types of first image data, second learning data in which second work position identification data, which is work position identification data indicating the second work position, is linked to multiple types of second image data, ..., nth learning data in which nth work position identification data, which is work position identification data indicating the nth work position, is linked to multiple types of nth image data.

[0060] Next, the trained model generation unit 420 generates a trained model for inferring a work position from new image data through machine learning using the acquired training data (step S102). Specifically, the trained model generation unit 420 inputs image data included in the training data into the input layer shown in Figure 5, and generates, as a trained model, a neural network in which the weights W1 and W2 are adjusted so that the result output from the output layer approaches a work position identifiable by the work position identification data linked to the image data. Then, the trained model output unit 430 outputs the trained model by transmitting the trained model to the personal computer 300 (step S103), and ends the process.

[0061] (Regarding the inference result data generation process according to the first embodiment) Next, a flowchart will be used to explain the operation of the data collection device 200, as an example of an inference device, to generate and output inference result data. When the data collection device 200 is turned on, it starts executing the inference result data generation process shown in Fig. 8. First, the data receiving unit 210, as an example of a trained model acquisition unit, receives the trained model from the personal computer 300 to acquire the trained model (step S201).

[0062] Next, the data receiving unit 210, which is an example of an inference data acquiring unit, receives new image data from the intelligent tool 100 and acquires the new image data as inference data (step S202). Next, the inference result data generating unit 240 inputs the inference data into the learned model and generates inference result data, which is data including score data indicating a score value for each type of work position (step S203). Then, the data collecting device 200 identifies data to be linked to the inference result data and transmitted to the external device 600 based on the setting data (step S204). The data transmitting unit 230 transmits the inference result data linked to the identified data, thereby outputting the inference result data (step S205), and the process ends.

[0063] As described above, according to the work management system 1 of the present embodiment, in the intelligent tool 100, the camera 130 captures an image of a work position where work is being performed on a workpiece using the intelligent tool 100. In the learning device 400, the learning data acquisition unit 410 acquires learning data including image data previously captured by the camera 130, and the trained model generation unit 420 generates a trained model for inferring a work position from new image data through machine learning using the learning data. In the data collection device 200 as an example of an inference device, the data receiving unit 210 as an example of an inference data acquisition unit acquires new image data captured by the camera 130 as inference data. In the data collection device 200, the inference result data generation unit 240 inputs the inference data into the trained model to generate inference result data that can identify the work position indicated by the new image data.

[0064] In this way, the work management system 1 according to this embodiment can identify the work position indicated by the new image data and can manage work performed on a workpiece as an object to be processed using the intelligent tool 100.

[0065] Here, there are various methods for image recognition processing using machine learning, broadly including object detection, classification, segmentation, and anomaly detection. Of these methods, it is natural for those skilled in the art to select an object recognition method in fields such as identifying the work position relative to a workpiece and the work target, as in the present embodiment. For example, the system disclosed in Patent Document 2 employs an object recognition method that recognizes objects in captured images through pattern recognition processing. However, as described above, the object recognition method employed by the system disclosed in Patent Document 2 has the problem that it is difficult to recognize objects in captured images if the resolution of the captured image is low, for example.

[0066] In contrast, in the work management system 1 according to this embodiment, the learning data includes first image data obtained by imaging toward the first work position where the first work is being performed, second image data obtained by imaging toward the second work position where the second work is being performed, ..., nth image data obtained by imaging toward the nth work position where the nth work is being performed. The trained model generation unit 420 then employs classification learning as machine learning, i.e., a classification method, to generate a trained model capable of classifying each image data from data obtained from around the work position.

[0067] In this way, in the work management system 1 according to the present embodiment, the trained model generation unit 420 learns to infer the work position based on data obtained from the periphery of the work position rather than on data obtained from the work position itself, rather than by recognizing objects at the work position. It is clear that the amount of information in the data obtained from the periphery of the work position is far greater than the amount of information in the data obtained from the work position itself. Therefore, the work management system 1 according to the present embodiment can generate inference result data that can identify the work position as one of the work positions better than the system disclosed in Patent Document 2, even when the image data has low resolution. As a result, the work management system 1 according to the present embodiment improves the recognition rate of work performed on a workpiece using the intelligent tool 100 compared to a work management system that does not generate a trained model capable of classifying each image data from data obtained from the periphery of the work position through classification learning.

[0068] Furthermore, according to the work management system 1 of this embodiment, in the learning device 400, the trained model generation unit 420 performs classification learning for each image data included in the learning data. In this way, the work management system 1 according to this embodiment can identify which of the work positions the new image data indicates, simply by learning the image data captured by the camera 130.

[0069] Furthermore, according to the work management system 1 of this embodiment, in the learning device 400, the learned model generation unit 420 performs classification learning based on the number of types of work position identification data linked by the personal computer 300, i.e., the number of categories. In this way, the work management system 1 according to this embodiment can perform classification learning more efficiently than a work management system that does not perform classification learning based on a predetermined number of categories. As a result, the work management system 1 according to this embodiment can improve the accuracy of identifying which of the work positions the work position indicated by new image data is, and further improve the recognition rate of work on a workpiece using the intelligent tool 100.

[0070] Furthermore, according to the work management system 1 of this embodiment, in the personal computer 300, the learning data generation unit 320 generates learning data by linking the processed image data, which has been subjected to image processing, together with the original image data and the work position identification data. Therefore, in the learning device 400, the learning data acquisition unit 410 acquires learning data that includes not only the original image data but also the processed image data.

[0071] In this way, the processed image data can be image data resulting from reproducing, through image processing, changes in an image according to the situation in which the camera 130 captures an image, that is, simulating the changes. Therefore, the trained model generation unit 420 can generate a trained model as a result of simulating various changes in the situation in which the camera 130 captures an image, by machine learning the training data including the processed image data. As a result, the work management system 1 according to this embodiment has improved accuracy in identifying which of the work positions the work position indicated by the new image data is, compared to a work management system that does not acquire training data including the processed image data, and further improves the recognition rate of work performed on a workpiece using the intelligent tool 100.

[0072] Furthermore, according to the work management system 1 of this embodiment, the data collection device 200 identifies data to be linked to the inference result data from various types of data, such as worker identification data, inference data, date and time data, tool-related data, and workpiece identification data, based on the setting data acquired from the personal computer 300. Then, the data transmission unit 230 transmits the inference result data and the identified data to the external device 600 in a state where they are linked together.

[0073] In this way, the external device 600 can perform data analysis using the work location identified from the inference result data and the work-related data linked to the inference result data. In particular, if the external device 600 is configured as a relational database, when inserting or updating the work location identified from the inference result data, the linked data can be used as identification information, making it possible to manage the acquired data more effectively than when the inference result data and various data are not linked.

[0074] Furthermore, according to the work management system 1 of this embodiment, the camera 130 is provided in the intelligent tool 100 and is supplied with power from the power supply unit 110, and the data transmission unit 132 of the camera 130 transmits the image data obtained by capturing the image to the data collection device 200 via wireless communication.

[0075] In this way, the power supply unit 110 of the intelligent tool 100 secures power for the camera 130, which consumes significantly less power than the power unit 120, so the intelligent tool 100 can realize the functions of the units 131 to 133 of the camera 130 with almost no loss in power supply to the power unit 120. In addition, in this way, the data collection device 200 can acquire data from the intelligent tool 100 via wireless communication. As a result, the work management system 1 according to this embodiment improves the efficiency of work by the intelligent tool 100 compared to a work management system that does not collect data from the intelligent tool via wireless communication.

[0076] Furthermore, according to the work management system 1 of this embodiment, the inference result data is data including score data indicating the score value for each type of work position calculated by the trained model. The inference result data indicates that, among the score values ​​exceeding a predetermined threshold, the work position with the largest score value is most likely to be the work position indicated by the new image data. By doing this, the work management system 1 of this embodiment has improved accuracy in identifying which of the work positions the work position indicated by new image data is, compared to a work management system whose inference result data does not include score data, and the recognition rate of work on a workpiece using the intelligent tool 100 is further improved.

[0077] [Embodiment 2] In the first embodiment, the data collection device 200 generates and outputs the inference result data as an inference device, but this is not limited to this. For example, the intelligent tool 100 may generate and output the inference result data as an inference device. Hereinafter, the work management system 1 according to the second embodiment will be described in detail with reference to FIG. 9. In the second embodiment, the configuration different from the first embodiment will be described, and the description of the same configuration as the first embodiment will be omitted to avoid redundancy.

[0078] (Regarding the work management system 1 according to the second embodiment) 9, in the work management system 1 according to this embodiment, the camera 130 of the intelligent tool 100, which is an example of an inference device, includes an inference result data generation unit 240, while the inference result data generation unit 240 is omitted from the data collection device 200. In addition, a data receiving unit 134 that receives data is added to the camera 130, and an imaging trigger data generation unit 250 that generates imaging trigger data, which will be described later, is added to the data collection device 200.

[0079] (Hardware Configuration of the Intelligent Tool 100 According to the Second Embodiment) In the intelligent tool 100, the control unit 51 shown in Fig. 3 functions as the inference result data generation unit 240 shown in Fig. 9 in accordance with the control program 59, and the transmission / reception unit 56 shown in Fig. 3 functions as the data receiving unit 134 shown in Fig. 9. Returning to Fig. 3, the intelligent tool 100 realizes the function of the inference result data generation unit 240 shown in Fig. 9 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes the inference result data generation step performed by the inference result data generation unit 240.

[0080] (Hardware Configuration of Data Collection Device 200 According to Second Embodiment) 3 functions as the imaging trigger data generation unit 250 shown in Fig. 9 in accordance with a control program 59. Returning to Fig. 3, the data collection device 200 realizes the function of the imaging trigger data generation unit 250 shown in Fig. 9 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the data collection device 200 executes the imaging trigger data generation step performed by the imaging trigger data generation unit 250.

[0081] (Details of Functional Configuration of Data Collection Device 200 According to Second Embodiment) The imaging trigger data generating unit 250 generates imaging trigger data that instructs the camera 130 to capture an image, based on the tool-related data received by the data receiving unit 210. For example, the imaging trigger data generating unit 250 generates imaging trigger data when it determines from the tool-related data that the operation of the power unit 120 has been completed. The imaging trigger data generating unit 250 may generate imaging trigger data based on the tool-related data and setting data. In this case, the setting data generating unit 330 in the personal computer 300 needs to generate setting data that includes data related to the generation of imaging trigger data, such as data indicating the conditions for generating imaging trigger data from the tool-related data and data indicating the timing for generating and transmitting the imaging trigger data.

[0082] The data transmission unit 230 transmits the imaging trigger data generated by the imaging trigger data generation unit 250 to the intelligent tool 100. In addition, the data transmission unit 230 transmits the trained model to the intelligent tool 100.

[0083] (Details of the functional configuration of the intelligent tool 100 as an example of an inference device) The data receiving unit 134, which is an example of a trained model acquiring unit, acquires the trained model by receiving the trained model from the data collecting device 200. The data receiving unit 134 also acquires imaging trigger data from the data collecting device 200. Note that the image data acquiring unit 131, which is an example of an inference data acquiring unit, acquires new image data when the data receiving unit 134 receives the imaging trigger data, thereby acquiring inference data.

[0084] The inference result data generation unit 240 inputs the inference data into the learned model to generate inference result data that can identify the work position. The data transmission unit 132 transmits the inference data, which is new image data acquired by the image data acquisition unit 131, and date and time data to the data collection device 200. The data transmission unit 132 also transmits the inference result data generated by the inference result data generation unit 240 to the data collection device 200.

[0085] As described above, according to the work management system 1 of the present embodiment, in the intelligent tool 100, the camera 130 includes the inference result data generation unit 240. In this way, in the work management system 1 according to the present embodiment, the camera 130 of the intelligent tool 100 can generate and output inference result data as an inference device. That is, in the present embodiment, unlike in the first embodiment, the intelligent tool 100 can generate and output inference result data as an inference device itself, without transmitting new image data as inference data to the data collection device 200 as an inference device. As a result, in the work management system 1 according to the present embodiment, the intelligent tool 100 can quickly perform inference as an inference device using new image data that it has acquired itself, without having to increase the communication speed of the network between the intelligent tool 100 and the inference device so that new image data can be quickly acquired.

[0086] Furthermore, according to the work management system 1 of this embodiment, the camera 130 is provided in the intelligent tool 100 and is supplied with power from the power supply unit 110, and the data transmission unit 132 of the camera 130 transmits the inference result data generated by the inference result data generation unit 240 to the data collection device 200 via wireless communication. By doing this, the power for the camera 130, which consumes significantly less power than the power unit 120, is secured from the power supply unit 110 of the intelligent tool 100, so that the intelligent tool 100 can realize the functions of each unit 131 to 134, 240 of the camera 130 with almost no loss in power supply to the power unit 120.

[0087] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, the imaging trigger data generation unit 250 generates imaging trigger data based on the tool-related data received from the intelligent tool 100, and the data transmission unit 230 transmits the imaging trigger data to the intelligent tool 100. Then, in the intelligent tool 100, when the camera 130 receives the imaging trigger data from the data collection device 200, the image data acquisition unit 131, which is an example of an inference data acquisition unit, acquires new image data as inference data.

[0088] In this way, the work management system 1 of this embodiment can instruct the timing of acquisition of inference data and inference by the intelligent tool 100 from the data collection device 200, which can send and receive data with the intelligent tool 100. In addition, the work management system 1 according to this embodiment has the same effects as the work management system 1 according to the first embodiment.

[0089] [Embodiment 3] In the first and second embodiments, the data collection device 200 and the intelligent tool 100 output the inference result data generated as inference devices to the external device 600, and the inference result data is used by the external device 600, but this is not limited to this, and the inference result data may also be used within the work management system 1. Hereinafter, the work management system 1 according to the third embodiment will be described in detail with reference to Figures 10 to 12. In the third embodiment, the configuration different from that of the first embodiment will be described, and the description of the same configuration as that of the first embodiment will be omitted to avoid redundancy.

[0090] (Regarding the Intelligent Tool 100 According to the Third Embodiment) As shown in FIG. 10, in the intelligent tool 100, the camera 130 further includes a data receiving unit 134 similar to that in the second embodiment, and a tool operation control unit 135 that controls the operation of the tool.

[0091] (Data Collection Device 200 According to Embodiment 3) The data collection device 200 further includes a work order determination unit 260 that determines the order of work, an activation data generation unit 270 that generates activation data that activates the operation of the intelligent tool 100, a sequence abnormality data generation unit 280 that generates sequence abnormality data that indicates an abnormality in the order of work, and a sequence abnormality data output unit 290 that outputs the sequence abnormality data.

[0092] (Hardware Configuration of the Intelligent Tool 100 According to the Third Embodiment) In the intelligent tool 100, the control unit 51 shown in Fig. 3 functions as a tool operation control unit 135 shown in Fig. 10 in accordance with a control program 59, and the transmission / reception unit 56 shown in Fig. 3 functions as a data receiving unit 134 shown in Fig. 10. Returning to Fig. 3, the intelligent tool 100 realizes the function of the tool operation control unit 135 shown in Fig. 10 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes the tool operation control steps performed by the tool operation control unit 135.

[0093] (Hardware Configuration of Data Collection Device 200 According to Embodiment 3) 3 functions as an operation sequence determination unit 260, an enabling data generation unit 270, and an out-of-sequence data generation unit 280 shown in FIG. 10 in accordance with a control program 59, and the display unit 55 shown in FIG. 3 functions as an out-of-sequence data output unit 290 shown in FIG. 10. Returning to FIG. 3, the data collection device 200 realizes the functions of the units 260 to 290 shown in FIG. 10 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the data collection device 200 executes an operation sequence determination step performed by the operation sequence determination unit 260 and an enabling data generation step performed by the validation data generation unit 270. For example, the data collection device 200 executes an out-of-sequence data generation step performed by the out-of-sequence data generation unit 280 and an out-of-sequence data output step performed by the out-of-sequence data output unit 290.

[0094] (Details of Functional Configuration of Data Collection Device 200 According to Embodiment 3) The inference result data generation unit 240 generates inference result data every time the data receiving unit 210 (an example of an inference data acquisition unit) acquires inference data, and stores the generated inference result data in the data storage unit 220. The data receiving unit 210 receives sequence data of work positions determined in advance based on the sequence of work on the workpieces as data necessary for managing the manufacturing from the manufacturing management device 500. The data collecting device 200 stores the sequence data received by the data receiving unit 210 in the data storage unit 220.

[0095] The work order determination unit 260 determines whether the order of work on the work is correct by comparing the historical data of the work position identified from the inference result data stored in the data storage unit 220 with the order data stored in the data storage unit 220. When the work order determination unit 260 determines that the order of work on the workpiece is correct, the activation data generation unit 270 generates activation data that activates the operation of the intelligent tool 100 to be used for the next work on the workpiece. When the activation data generation unit 270 generates the activation data, the data transmission unit 230 transmits the activation data to the intelligent tool 100 to be used for the next work.

[0096] The sequence error data generating unit 280 generates sequence error data when the work order determining unit 260 determines that the order of work performed on the workpieces is incorrect. The out-of-sequence data output unit 290 outputs the out-of-sequence data by, for example, displaying an out-of-sequence image shown in FIG. 11 on the display unit 55 or by outputting an alarm sound.

[0097] (Details of the Functional Configuration of the Intelligent Tool 100 According to the Third Embodiment) The data receiving unit 134 receives validation data from the data collecting device 200 . The tool operation control unit 135 disables the output operation by the power unit 120 until the data receiving unit 134 receives the enabling data, and controls the output operation by the power unit 120 to be enabled when the data receiving unit 134 receives the enabling data.

[0098] (Regarding the Work Sequence Determination Process According to the Third Embodiment) Next, the operation of the data collection device 200 to determine the order of work and transmit validation data will be described using a flowchart. When the inference result data generation unit 240 generates inference result data and stores it in the data storage unit 220, the data collection device 200 starts executing the work order determination process shown in Fig. 12. First, the work order determination unit 260 identifies history data of work locations from multiple types of inference result data stored in the data storage unit 220 (step S301), and acquires the order data stored in the data storage unit 220 (step S302).

[0099] Next, the work order determination unit 260 compares the history data with the order data to determine whether the order of the work on the workpiece is correct (step S303). If the work order determination unit 260 determines that the order of the work is correct (step S303; Y), the validation data generation unit 270 generates validation data (step S304), the data transmission unit 230 transmits the validation data to the intelligent tool 100 to be used for the next work (step S305), and the process ends. On the other hand, if the work order determination unit 260 determines that the order of the work is incorrect (step S303; N), the sequence abnormality data generation unit 280 generates sequence abnormality data (step S306), and the sequence abnormality data output unit 290 outputs the sequence abnormality data by, for example, displaying the sequence abnormality image shown in FIG. 11 on the display unit 55 or outputting an alarm sound (step S307), and the process ends.

[0100] As described above, according to the work management system 1 of this embodiment, in the data collection device 200, the inference result data generation unit 240 generates inference result data each time the data receiving unit 210 (an example of an inference data acquisition unit) acquires inference data, and stores the generated inference result data in the data storage unit 220. Furthermore, the work order determination unit 260 identifies work position history data from the multiple types of inference result data stored in the data storage unit 220. Then, the work order determination unit 260 compares the identified work position history data with work position order data that is predetermined based on the order of work on the workpieces stored in the data storage unit 220, thereby determining whether the order of work on the workpieces is correct. In this way, the work management system 1 according to this embodiment can use the inference result data to determine whether the order of work on the work is correct or not.

[0101] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, when the work order determination unit 260 determines that the order of work on the workpiece is correct, the activation data generation unit 270 generates activation data, and the data transmission unit 230 transmits the activation data to the intelligent tool 100 to be used for the next work. Then, in the intelligent tool 100, the tool operation control unit 135 controls to activate the output operation of the power unit 120 when the data receiving unit 134 receives the activation data. In this way, the work management system 1 according to the present embodiment can prevent work that has been carried out in an incorrect order from being continued.

[0102] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, when the work order determination unit 260 determines that the order of work on the work is incorrect, the sequence abnormality data generation unit 280 generates sequence abnormality data, and the sequence abnormality data output unit 290 outputs the sequence abnormality data. In this way, the work management system 1 according to the present embodiment can notify the system participants, such as workers and users, that work has been performed in an incorrect order. In addition, the work management system 1 according to this embodiment has the same effects as the work management system 1 according to the first embodiment.

[0103] [Example of change] In the above first to third embodiments, a worker performs work on a workpiece using the intelligent tool 100, but the present invention is not limited to this. For example, a robot may perform work on a workpiece using the intelligent tool 100.

[0104] As in the above-mentioned embodiments 1 to 3, it is preferable to use the learning device 400 as a cloud server for machine learning, but this is not limited to this. For example, the data collection device 200 and the personal computer 300 may include the functions of each of the units 410 to 430 of the learning device 400, and these computer devices may operate as learning devices.

[0105] In the above-described first to third embodiments, the functions of the inference device are included in the intelligent tool 100 or the data collection device 200, and the intelligent tool 100 or the data collection device 200 operates as the inference device, but this is not limiting. For example, the functions of the inference device may be included in the personal computer 300 or the learning device 400, and the personal computer 300 or the learning device 400 may operate as the inference device. In addition, if the learning device 400 also operates as the inference device, the inference by the inference device will also be implemented as a cloud system.

[0106] In the above-described first to third embodiments, the case where supervised learning is applied to the learning algorithm used by the trained model generation unit 420 has been described, but the present invention is not limited to this. As for the learning algorithm, reinforcement learning, unsupervised learning, semi-supervised learning, or the like can also be applied in addition to supervised learning.

[0107] The trained model generation unit 420 may also learn the work position indicated by new image data according to training data created for the same work at multiple types of work sites. It is also possible to add or remove work positions for which training data is collected during the process. Furthermore, the learning device 400 that has trained to infer the work position indicated by new image data for a certain work may be applied to a different work, and the work position indicated by the new image data for the different work may be re-trained and updated.

[0108] In addition, the learning algorithm used in the trained model generation unit 420 can be deep learning, which learns to extract the features themselves, or machine learning can be performed according to other known methods, such as genetic programming, functional logic programming, support vector machines, etc.

[0109] In the above-described first to third embodiments, the work position indicated by new image data is inferred using a trained model learned about work performed at a work site managed by the work management system 1, but this is not limiting. For example, a trained model generated by machine learning in an external system for the same work performed at another work site not managed by the work management system 1 may be obtained, and the acquired trained model may be used to infer the work position indicated by new image data.

[0110] In the first to third embodiments, the intelligent tool 100 and the data collecting device 200 are connected via a wireless LAN. However, this is not limiting. For example, data may be transmitted and received via wireless communication using other wireless communication standards, such as Bluetooth (registered trademark). In this case, at least the intelligent tool 100 and the data collecting device 200 need to be paired to establish a state in which they can communicate one-to-one. Furthermore, as in the first to third embodiments, it is preferable that the intelligent tool 100 and the data collecting device 200 can transmit and receive data via wireless communication. However, this is not limiting. For example, the intelligent tool 100 and the data collecting device 200 may transmit and receive data via wired communication, such as a wired LAN.

[0111] As in the above embodiments 1 to 3, it is preferable that the camera 130 is provided with an illumination light output device to increase the brightness of the captured image, but this is not limited to this as long as the image can be captured by the imaging sensor, and the illumination light output device may be omitted from the camera 130.

[0112] In the above-described first and third embodiments, camera 130 automatically acquires image data when it detects that power unit 120 has completed its operation, and in the above-described second embodiment, it acquires image data when it receives imaging trigger data, but the conditions for acquiring image data are not limited to these. For example, camera 130 may acquire image data when an operator operates an imaging switch provided on camera 130. In other words, camera 130 may acquire image data manually by an operator operating the imaging switch.

[0113] As in the above-described first to third embodiments, the learning data generation unit 320 preferably generates a folder for each work position as work position identification data and includes image data of the corresponding work position in the folder, but is not limited to this. For example, learning data that associates the work position identification data with the image data may be generated by adding the work position identification data to the image data.

[0114] In the first and third embodiments, the camera 130 acquires image data when it detects the completion of the operation of the power unit 120. However, this is not limited to this. Multiple types of image data may be acquired from the start of operation of the power unit 120 to the completion of operation. In this case, the data for inference may be multiple types of new image data, and the inference device may input all image data as inference data into the trained model and identify the work position with the highest score value as the work position indicated by the inference data. Furthermore, in this case, the data for inference may be the image data that is most suitable for inference from the multiple types of new image data. By doing this, even if the acquired image data includes low contrast and sharpness due to the time required for various adjustments such as autofocus and automatic exposure adjustment, the image data that is statistically considered to have the best contrast and sharpness can be used as the data for inference.

[0115] In the above-described first to third embodiments, the inference device infers that the work position indicated by the new image data cannot be identified if the maximum score value indicated by the score data included in the inference result data does not exceed a threshold value. However, this is not limited to this. For example, the inference device may omit the process of determining whether the maximum score value exceeds a threshold value, and infer that the work position with the maximum score value is the work position indicated by the new image data.

[0116] In the above-described first to third embodiments, the data collection device 200 updates various data that may be linked to inference result data based on the setting data to the latest data when the intelligent tool 100 and the manufacturing control device 500 transmit the data. However, the present invention is not limited to this. For example, the data collection device 200 may request the intelligent tool 100 and the manufacturing control device 500 to transmit various data at a predetermined polling interval, thereby updating the various data to the latest data at the polling interval. In this case, for example, the personal computer 300 may generate setting data including collection condition data indicating conditions, such as the polling interval, for causing the data collection device 200 to collect the various data described above, and transmit the setting data to the data collection device 200 to set conditions for the data collection device 200 to update the various data to the latest data.

[0117] In the above-described first to third embodiments, the data collection device 200 transmits the inference result data and various data acquired from the intelligent tool 100 or the manufacturing management device 500 to the external device 600 in a linked state. However, the present invention is not limited to this, and the data does not have to be transmitted to the external device 600. In this case, the data collection device 200 may store the data in which the inference result data and various data are linked in the data storage unit 220 in a data format suitable for data analysis, such as CSV (Comma Separated Value). In this way, the data collection device 200, like the external device 600, can perform data analysis using the operation position identified from the inference result data and the data related to the operation linked to the inference result data.

[0118] In the third embodiment, when the data collection device 200 determines that the order of operations on the workpiece is correct, it generates validation data and transmits it to the intelligent tool 100 to be used for the next operation. However, the processing performed when it is determined that the order of operations on the workpiece is correct is not limited to this. For example, when the data collection device 200 determines that the order of operations on the workpiece is correct, it may generate data indicating that the order of operations was correct and transmit the data to the intelligent tool 100 that performed the operations. In this case, the data collection device 200 may determine whether the state of operation completion is appropriate based on the tool-related data, and if not, transmit data indicating that to the intelligent tool 100. For example, when it determines that a screw is not sufficiently tightened based on data indicating a torque value and data indicating a torque waveform, it may transmit data indicating that to the intelligent tool 100. In this way, the intelligent tool 100 can recognize the information and allow the worker to appropriately perform the operation at the inferred operation position.

[0119] In the third embodiment, the data collection device 200 outputs the sequence abnormality data by displaying the sequence abnormality image shown in FIG. 11 on the display unit 55 or by outputting an alarm sound. However, the output of the sequence abnormality data is not limited to this. For example, the sequence abnormality data may be transmitted to another device, such as the intelligent tool 100 that performed the work, the intelligent tool 100 that will perform the next work, the manufacturing management device 500, or the external device 600, i.e., output externally. In this way, the user of the other device that receives the sequence abnormality data can recognize that the work was performed in an incorrect order.

[0120] The core processing sections of the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400, each of which includes the control unit 51, the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, the transmission / reception unit 56, and the internal bus 50, can be realized using a general computer system rather than a dedicated system. For example, a computer program for executing the above operations may be stored and distributed on a computer-readable recording medium, such as a flexible disk or a DVD-ROM (Read-Only Memory), and the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400 may be configured to execute the above processes by installing the computer program on a computer. Alternatively, the computer program may be stored in a storage device of a server device on a communication network and downloaded by a general computer system to configure the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400.

[0121] In addition, when the functions of the intelligent tool 100, the data collection device 200, the personal computer 300, and the learning device 400 are realized by sharing the functions of an OS (Operating System) and an application program, or by collaboration between the OS and an application program, only the application program portion may be stored in a recording medium or storage device.

[0122] It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and provided via the network. The computer program may then be started and executed under the control of the OS in the same way as other application programs, thereby performing the above-mentioned processing.

[0123] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure. [Explanation of symbols]

[0124] 1...work management system, 50...internal bus, 51...control unit, 52...main memory unit, 53...external memory unit, 54...operation unit, 55...display unit, 56...transmission / reception unit, 59...control program, 100...intelligent tool, 101...first intelligent tool, 102...second intelligent tool, 109...ninth intelligent tool, 110...power supply unit, 120...power unit, 130...camera, 131...image data acquisition unit, 132...data transmission unit, 133...tool state monitoring unit, 134...data reception unit, 135...tool operation control unit, 200...data collection device, 210...data reception unit , 220...data storage unit, 230...data transmission unit, 240...inference result data generation unit, 250...imaging trigger data generation unit, 260...work sequence determination unit, 270...validation data generation unit, 280...sequence abnormality data generation unit, 290...sequence abnormality data output unit, 300...personal computer, 310...data receiving unit, 320...learning data generation unit, 330...setting data generation unit, 340...data transmission unit, 400...learning device, 410...learning data acquisition unit, 420...trained model generation unit, 430...trained model output unit, 500...manufacturing management equipment, 600...external device.

Claims

1. A work management system that manages work on a workpiece as a processing object using a tool, a learning data acquisition unit that acquires learning data including image data previously captured by an imaging member that captures an image toward a work position where work is being performed on the workpiece using the tool; and a trained model generation unit that generates a trained model for inferring the work position from new image data by machine learning using the training data; an inference data acquisition unit that acquires the new image data obtained by the imaging member as inference data; an inference result data generation unit that inputs the acquired inference data into the trained model each time the inference data acquisition unit acquires the inference data, and generates inference result data that can identify the work position; Equipped with The learning data includes first image data obtained by capturing an image toward a first work position where a first task is being performed, and second image data obtained by capturing an image toward a second work position where a second task different from the first task is being performed, the trained model generation unit generates the trained model capable of classifying the first image data and the second image data from data obtained from around the work position by classification learning as the machine learning; The learning data includes original image data obtained by capturing an image in advance using the imaging member, and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging member capturing an image from a first angle, and second original image data obtained by the imaging member capturing an image from a second angle different from the first angle; the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce changes in the situation captured by the imaging member; a work order determination unit that determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with the work position order data that is predetermined based on the order of work on the workpiece; and a tool operation control unit that enables an operation of the tool to be used for a next operation on the workpiece when the operation order determination unit determines that the order of operations on the workpiece is correct; and Further provided with The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, the tool operation control unit, when the tool used in the previous work determined by the work order determination unit to be in the correct order is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, enables the operation of the other tool. Work management system.

2. The trained model generation unit performs the classification learning for each of the image data included in the training data. The work management system according to claim 1 .

3. The trained model generation unit performs the classification learning based on a predetermined number of categories. The work management system according to claim 1 .

4. the imaging member includes the inference result data generation unit, The work management system according to claim 1 .

5. The worker performs the work, a data transmission unit that transmits to an external device at least one of worker identification data that can identify the worker, date and time data that indicates the date and time when the new image data that becomes the inference data was acquired, workpiece identification data that can identify the workpiece, tool-related data that relates to the tool, and the inference data, and the inference result data that is associated with the one piece of data; Further comprising: The work management system according to claim 1 .

6. a data collection device that collects data related to management of work on the workpiece; Further provided with the imaging member is supplied with power from the tool; the imaging member includes a data transmission unit that transmits the image data obtained by imaging to the data collection device by wireless communication; The work management system according to claim 1 .

7. a data collection device that collects data related to management of work on the workpiece; Further provided with the imaging member is supplied with power from the tool; the imaging member includes a data transmission unit that transmits the inference result data generated by the inference result data generation unit to the data collection device by wireless communication; The work management system according to claim 4 .

8. The inference result data includes score data indicating a score value for each predetermined classification calculated by the trained model, The work position that can be identified from the inference result data is a position that is identified by one classification in which the score value is the maximum value among values ​​that exceed a predetermined threshold value. The work management system according to claim 1 .

9. the imaging member captures images a plurality of times from the start of a task on the workpiece using the tool until the task is completed; the inference data acquisition unit acquires the inference data based on the plurality of types of new image data obtained by the imaging member capturing images of the one task a plurality of times, The work management system according to claim 1 .

10. a learning data acquisition unit that acquires learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a processing object using a tool, the learning data including first image data obtained by capturing an image toward the first work position where a first task is being performed, and second image data obtained by capturing an image toward a second work position where a second task different from the first task is being performed; and a trained model generation unit that generates a trained model for inferring the work position from new image data by machine learning using the training data, the trained model being capable of classifying the first image data and the second image data by classification learning as the machine learning; an inference data acquisition unit that acquires the new image data obtained by the imaging unit as inference data; an inference result data generation unit that inputs the acquired inference data into the trained model each time the inference data acquisition unit acquires the inference data, and generates inference result data that can identify the work position; Equipped with the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; a work order determination unit that determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with the work position order data that is predetermined based on the order of work on the workpiece; and a tool operation control unit that enables an operation of the tool to be used for a next operation on the workpiece when the operation order determination unit determines that the order of operations on the workpiece is correct; and Further provided with The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, the tool operation control unit, when the tool used in the previous work determined by the work order determination unit to be in the correct order is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, enables the operation of the other tool. Learning device.

11. a trained model acquisition unit that acquires a trained model for inferring the work position from new image data generated by machine learning using learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a work object using a tool, the training data including first image data obtained by capturing an image toward the first work position where a first work is being performed and second image data obtained by capturing an image toward a second work position where a second work different from the first work is being performed, the trained model being capable of classifying the first image data and the second image data by classification learning as the machine learning; and an inference data acquisition unit that acquires the new image data obtained by the imaging unit as inference data; an inference result data generation unit that inputs the acquired inference data into the trained model each time the inference data acquisition unit acquires the inference data, and generates inference result data that can identify the work position; Equipped with the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; a work order determination unit that determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with the work position order data that is predetermined based on the order of work on the workpiece; and a tool operation control unit that enables an operation of the tool to be used for a next operation on the workpiece when the operation order determination unit determines that the order of operations on the workpiece is correct; and Further provided with The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, the tool operation control unit, when the tool used in the previous work determined by the work order determination unit to be in the correct order is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, enables the operation of the other tool. Reasoning device.

12. a learning data acquisition step in which the computer acquires learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a workpiece using a tool, the learning data including first image data obtained by capturing an image toward the first work position where a first task is being performed, and second image data obtained by capturing an image toward a second work position where a second task different from the first task is being performed; a trained model generation step in which the computer generates a trained model for inferring the work position from new image data by machine learning using the learning data, the trained model being capable of classifying the first image data and the second image data by classification learning as the machine learning; an inference data acquisition step in which the computer acquires the new image data obtained by capturing an image with the imaging unit as inference data; an inference result data generation step in which the computer inputs the acquired inference data into the trained model each time the computer acquires the inference data in the inference data acquisition step to generate inference result data that can identify the work position; Including, the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; a work order determination step in which the computer determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated in the inference result data generation step with the work position order data predetermined based on the order of work on the workpiece; a tool operation control step of enabling an operation of the tool to be used for a next operation on the workpiece when the computer determines in the operation order determination step that the order of operations on the workpiece is correct; Further provided with The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, In the tool operation control step, when the tool used in the previous work determined to be in the correct order in the work order determination step is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, the computer enables the operation of the other tool. Control method.

13. a learned model acquisition step in which a computer acquires a learned model for inferring the work position from new image data generated by machine learning using learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a workpiece using a tool, the learning data including first image data obtained by capturing an image toward the first work position where a first work is being performed and second image data obtained by capturing an image toward a second work position where a second work different from the first work is being performed, the learned model being capable of classifying the first image data and the second image data by classification learning as the machine learning; an inference data acquisition step in which the computer acquires the new image data obtained by capturing an image with the imaging unit as inference data; an inference result data generation step in which the computer inputs the acquired inference data into the trained model each time the computer acquires the inference data in the inference data acquisition step to generate inference result data that can identify the work position; Including, the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; a work order determination step in which the computer determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated in the inference result data generation step with the work position order data predetermined based on the order of work on the workpiece; a tool operation control step of enabling an operation of the tool to be used for a next operation on the workpiece when the computer determines in the operation order determination step that the order of operations on the workpiece is correct; Further provided with The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, In the tool operation control step, when the tool used in the previous work determined to be in the correct order in the work order determination step is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, the computer enables the operation of the other tool. Control method.

14. Computer, a learning data acquisition unit that acquires learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a processing object using a tool, the learning data including first image data obtained by capturing an image toward the first work position where a first work is being performed, and second image data obtained by capturing an image toward a second work position where a second work different from the first work is being performed; a trained model generation unit that generates a trained model for inferring the work position from new image data by machine learning using the training data, the trained model being capable of classifying the first image data and the second image data by classification learning as the machine learning; an inference data acquisition unit that acquires the new image data obtained by the imaging unit as inference data; an inference result data generation unit that inputs the acquired inference data into the trained model each time the inference data acquisition unit acquires the inference data, and generates inference result data that can identify the work position; It functions as the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; The computer a work order determination unit that determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with the work position order data that is predetermined based on the order of work on the workpiece; a tool operation control unit that enables an operation of the tool to be used for a next operation on the workpiece when the operation order determination unit determines that the order of operations on the workpiece is correct; It further functions as The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, the tool operation control unit, when the tool used in the previous work determined by the work order determination unit to be in the correct order is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, enables the operation of the other tool. program.

15. Computer, a trained model acquisition unit that acquires a trained model for inferring the work position from new image data generated by machine learning using learning data including image data previously captured by an imaging unit that captures an image toward a work position where a work is being performed on a workpiece as a work object using a tool, the training data including first image data obtained by capturing an image toward the first work position where a first work is being performed and second image data obtained by capturing an image toward a second work position where a second work different from the first work is being performed, the trained model being capable of classifying the first image data and the second image data by classification learning as the machine learning; an inference data acquisition unit that acquires the new image data obtained by the imaging unit as inference data; an inference result data generation unit that inputs the acquired inference data into the trained model each time the inference data acquisition unit acquires the inference data, and generates inference result data that can identify the work position; It functions as the learning data includes original image data obtained by capturing an image in advance by the imaging unit and processed image data obtained by performing image processing on the original image data, the original image data includes at least first original image data obtained by the imaging unit capturing an image from a first angle, and second original image data obtained by the imaging unit capturing an image from a second angle different from the first angle, the processed image data includes at least first processed image data obtained by performing image processing on the first original image data in order to reproduce a change in a situation in which the image capturing unit captures an image; The computer a work order determination unit that determines whether the order of work on the workpiece is correct by comparing the work position history data identified from the multiple types of inference result data generated by the inference result data generation unit with the work position order data that is predetermined based on the order of work on the workpiece; a tool operation control unit that enables an operation of the tool to be used for a next operation on the workpiece when the operation order determination unit determines that the order of operations on the workpiece is correct; It further functions as The tools include a first tool and a second tool having a different purpose from the first tool based on a function realized by an output operation performed by a power unit, the tool operation control unit, when the tool used in the previous work determined by the work order determination unit to be in the correct order is one of the first tool and the second tool, and the tool to be used in the next work is the other of the first tool and the second tool, enables the operation of the other tool. program.

Images