Work management system, training device, inference device, control method, and program

Abstract

A training data acquisition unit (410) of a work management system (1) acquires training data including image data captured by a camera (130) in advance, a trained model generation unit (420) generates a trained model by machine learning using the training data, and an inference result data generation unit (240) inputs inference data to the trained model to generate inference result data enabling identification of a work position shown by new image data. The training data includes first image data captured toward a first work position, and second image data captured toward a second work position, and the trained model generation unit (420) uses classification learning to generate a trained model capable of classifying each image data on the basis of data obtained from the periphery of a work position.

Description

Work management system, learning device, inference device, control method and program

[0001] The present disclosure relates to a work management system, a learning device, an inference device, a control method, and a program.

[0002] 2. Description of the Related Art There is known a work management system that uses a camera to identify and manage work performed by a worker using a tool on a workpiece as an object to be processed at a work site such as a factory.

[0003] Patent Document 1 discloses a work management system that calculates the three-dimensional position of a marker attached to a tool using image data obtained when the marker is captured by a camera, and compares the calculated three-dimensional position with the three-dimensional position of the tool during a predetermined work operation to determine whether the work is correct or incorrect. In the system disclosed in Patent Document 1, the camera is installed at a position away from the tool, and when a worker works in a complicated place, there are many blind spots for the camera, so if the marker cannot be captured, the three-dimensional position of the tool cannot be calculated and it is not possible to determine whether the work is correct or incorrect.

[0004] In response to this, Patent Document 2 discloses a tool system that identifies a set work target, which is a work target on which a tool is set, among multiple work targets based on an image captured by an imaging unit provided on the tool. In Patent Document 2, the imaging unit is attached to the tip of a conductor and is configured so that a socket attached to the output shaft of the tool falls within the imaging range. Furthermore, the system disclosed in Patent Document 2 performs pattern recognition processing on video-format images captured by the imaging unit when a user works with the tool, using a reference image including multiple frames corresponding to multiple work targets as template data, thereby identifying the set work target. Specifically, in the pattern recognition processing, the captured image is input into a trained model created by machine learning using the template data, and the set work target is identified by recognizing objects reflected in the captured image.

[0005] JP 2013-132736 A JP 2021-178397 A

[0006] In the system disclosed in Patent Document 1, the three-dimensional position of a marker is calculated using image data obtained from a camera to determine whether the task is correct or incorrect, so there is a risk that discrimination will be difficult if, for example, the resolution of the image data is low or the lens is out of focus and the boundary between the image of the marker and other images becomes unclear. Similarly, in the system disclosed in Patent Document 2, the object shown in the captured image is recognized by pattern recognition processing, so there is a risk that recognition of the object shown will be difficult if, for example, the resolution of the captured image is low or the lens is out of focus and the boundary between the image of the object shown and other images becomes unclear.

[0007] The present disclosure has been made in consideration of the above-described circumstances, and aims to improve the recognition rate of work performed on a workpiece using a tool.

[0008] To achieve the above object, a work management system according to the present disclosure is a work management system that manages work on a workpiece (a target object) using a tool, and includes: a training data acquisition unit that acquires training data including image data previously captured by an imaging member that captures images of a work position where work is being performed on the workpiece using the tool; a trained model generation unit that generates a trained model for inferring the work position from new image data through machine learning using the training data; an inference data acquisition unit that acquires the new image data captured by the imaging member as inference data; and an inference result data generation unit that inputs the inference data into the trained model to generate inference result data that can identify the work position. The training data includes first image data obtained by capturing images of a work position where a first task is being performed and second image data obtained by capturing images of a work position where a second task different from the first task is being performed, and the trained model generation unit generates a trained model that can classify the first image data and the second image data from data obtained from around the work position through classification learning as machine learning.

[0009] According to the present disclosure, the trained model generation unit generates a trained model capable of classifying first image data and second image data from data obtained from the vicinity of the work position through classification learning, and the inference result data generation unit can input inference data to the trained model to generate inference result data that can identify whether the work position is the first work position or the second work position. As a result, the work management system according to the present disclosure has a higher recognition rate for work performed using tools on workpieces than a work management system that does not generate a trained model capable of classifying first image data and second image data through classification learning.

[0010] FIG. 1 is a block diagram showing the hardware configuration of each device according to the first embodiment; FIG. 2 is an explanatory diagram of image data and date and time data according to the first embodiment; FIG. 3 is an explanatory diagram of learning data according to the first embodiment; FIG. 4 is an explanatory diagram of a neural network according to the first embodiment; FIG. 5 is a diagram showing an example of display of score data of inference result data according to the first embodiment;

[0011] Hereinafter, a work management system, a learning device, an inference device, a control method, and a program according to embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are designated by the same reference numerals.

[0012] [First Embodiment] (Regarding the Work Management System 1 According to the First Embodiment) A work management system 1 according to the first embodiment of the present disclosure is a system for managing work on a workpiece, which is an object to be processed at a manufacturing site such as a factory. As shown in Fig. 1 , the work management system 1 includes an intelligent tool 100 as an example of a tool used by an operator, and a data collection device 200 that collects data from the intelligent tool 100. The work management system 1 also includes a personal computer 300 installed with software capable of realizing an engineering environment used by the user, and a learning device 400 that learns about a work position, which is a position where the operator is performing work on the workpiece using the intelligent tool 100.

[0013] The intelligent tool 100 and the data collecting device 200 can transmit and receive data via a wireless LAN (Local Area Network) using a wireless communication standard such as Wi-Fi (registered trademark). The data collecting device 200 and the personal computer 300 can transmit and receive data via a wired LAN. The personal computer 300 and the learning device 400 can transmit and receive data via the Internet.

[0014] (Regarding the Intelligent Tool 100 According to the First Embodiment) The intelligent tool 100 is a tool equipped with the functionality of a computer device capable of performing data processing using a program. The intelligent tool 100 includes multiple types of intelligent tools with different uses, such as a first intelligent tool 101, a second intelligent tool 102, ..., and a ninth intelligent tool 109. Note that the intelligent tools 101 to 109 differ only in the function of the power unit, which will be described later, but are otherwise similar in function. For this reason, in order to reduce redundant explanation, "each of the intelligent tools 101 to 109" will hereinafter be simply referred to as "the intelligent tool 100."

[0015] As shown in FIG. 2 , the intelligent tool 100 includes a power supply unit 110 that supplies power to the tool, a power unit 120 that performs output operations for various uses of the tool, an imaging unit that captures images of the work position where the power unit 120 performs the output operation, and a camera 130 as an example of an imaging member. The power supply unit 110 is a power supply member such as a lithium-ion battery, and the power unit 120 and the camera 130 are operable with power supplied from the power supply unit 110. In this embodiment, the camera 130 includes, for example, an imaging sensor and an illumination light output device (not shown). The camera 130 may also be provided with various mechanisms, such as an autofocus mechanism and an automatic exposure adjustment mechanism (not shown). The camera 130 also includes the functions of the computer device described above. Specifically, the camera 130 includes an image data acquisition unit 131 that acquires image data by capturing images, a data transmission unit 132 that transmits the data, and a tool status monitoring unit 133 that monitors the operating status of the tool.

[0016] (Regarding the Data Collection Device 200 According to the First Embodiment) The data collection device 200, which is an example of an inference device, is a computer device serving as a so-called database server that accumulates data collected from the intelligent tool 100. The data collection device 200 includes a data receiving unit 210 that receives data, a data storage unit 220 that stores data, a data transmitting unit 230 that transmits data, and an inference result data generating unit 240 that generates inference result data (described later) using a trained model (described later).

[0017] (Regarding the personal computer 300 according to embodiment 1) The personal computer 300 includes a data receiving unit 310 that receives data, a learning data generating unit 320 that generates learning data described below, a setting data generating unit 330 that generates various setting data for setting the operation of the work management system 1, and a data transmitting unit 340 that transmits data.

[0018] (Regarding the Learning Device 400 According to Embodiment 1) The learning device 400 is a computer device serving as a cloud server for machine learning, equipped with, for example, a computing device such as a central processing unit (CPU) or a graphics processing unit (GPU) capable of high-speed data processing. The learning device 400 includes a training data acquisition unit 410 that acquires training data (described later), a trained model generation unit 420 that generates a trained model (described later), and a trained model output unit 430 that outputs the trained model. Note that, because the generation and output of the trained model only requires the prior acquisition of training data, the load on the network can be reduced even if the amount of information in the training data required for machine learning increases. Therefore, by using the learning device 400 as a cloud server, the machine learning function can be implemented as a cloud system, which is suitable for actually building the work management system 1.

[0019] (Hardware Configuration of the Intelligent Tool 100 According to the First Embodiment) The hardware configuration of the intelligent tool 100 will be described below. In this embodiment, the functions of the intelligent tool 100 as a computer device are included in the camera 130. Therefore, in the following description, the "intelligent tool 100" can be read as the "camera 130."

[0020] 3, the intelligent tool 100 includes a control unit 51 that executes processing in accordance with a control program 59. The control unit 51 includes a CPU. In accordance with the control program 59, the control unit 51 functions as an image data acquisition unit 131 and a tool state monitoring unit 133 shown in FIG.

[0021] 3, the intelligent tool 100 includes a main memory unit 52 into which a control program 59 is loaded and which is used as a work area for the control unit 51. The main memory unit 52 includes a RAM (Random Access Memory).

[0022] The intelligent tool 100 also includes an external storage unit 53 that pre-stores a control program 59. The external storage unit 53 supplies data to be stored by the program to the control unit 51 in accordance with instructions from the control unit 51, and stores data supplied from the control unit 51. The external storage unit 53 includes a non-volatile memory such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0023] The intelligent tool 100 also includes an operation unit 54 that is operated by a user. Information input via the operation unit 54 is supplied to the control unit 51. The operation unit 54 includes information input components such as a keyboard, a mouse, and a touch panel.

[0024] The intelligent tool 100 may include a display unit 55 that displays information input via the operation unit 54 and information output by the control unit 51. In this case, the display unit 55 may include a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0025] The intelligent tool 100 also includes a transceiver 56 for transmitting and receiving information. The transceiver 56 includes information communication components such as a communication network terminal connected to a network and a wireless communication device. The transceiver 56 functions as the data transmitter 132 shown in FIG. 2.

[0026] Returning to FIG. 3, in the intelligent tool 100 , the main memory unit 52 , the external memory unit 53 , the operation unit 54 , the display unit 55 and the transmission / reception unit 56 are all connected to the control unit 51 via the internal bus 50 .

[0027] 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes an image data acquisition step performed by the image data acquisition unit 131, a data transmission step performed by the data transmission unit 132, and a tool state monitoring step performed by the tool state monitoring unit 133.

[0028] (Hardware Configuration of Data Collection Device 200 According to Embodiment 1) As shown in Fig. 3, the data collection device 200 includes the above-described control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the inference result data generation unit 240 shown in Fig. 2 in accordance with a control program 59. The external memory unit 53 functions as the data storage unit 220 shown in Fig. 2. The transmission / reception unit 56 functions as the data reception unit 210 and data transmission unit 230 shown in Fig. 2.

[0029] 3 , the data collection device 200 realizes the functions of the above-described units 210 to 240 shown in FIG. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the data collection device 200 executes a trained model acquisition step performed by a trained model acquisition unit and a data receiving unit 210 as an example of an inference data acquisition unit, and a data receiving step as an example of an inference data acquisition step. Also, for example, the data collection device 200 executes a data storage step performed by a data storage unit 220, a data transmission step as an example of an inference result data output step performed by a data transmission unit 230 as an example of an inference result data output unit, and an inference result data generation step performed by an inference result data generation unit 240.

[0030] (Hardware Configuration of Personal Computer 300 According to First Embodiment) As shown in Fig. 3, similar to the data collection device 200, the personal computer 300 also includes the above-described control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the learning data generation unit 320 and the setting data generation unit 330 shown in Fig. 2 in accordance with a control program 59. The transmission / reception unit 56 functions as the data reception unit 310 and the data transmission unit 340 shown in Fig. 2.

[0031] 3, the personal computer 300 realizes the functions of the above-mentioned units 310 to 340 shown in Fig. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the personal computer 300 executes a data reception step performed by the data reception unit 310, a learning data generation step performed by the learning data generation unit 320, a setting data generation step performed by the setting data generation unit 330, and a data transmission step performed by the data transmission unit 340.

[0032] (Hardware Configuration of Learning Device 400 According to Embodiment 1) As shown in Fig. 3 , like the data collection device 200 and the personal computer 300, the learning device 400 also includes the above-described control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, and transmission / reception unit 56. The control unit 51 functions as the trained model generation unit 420 shown in Fig. 2 in accordance with a control program 59. The transmission / reception unit 56 functions as the training data acquisition unit 410 and trained model output unit 430 shown in Fig. 2 .

[0033] 3, the learning device 400 realizes the functions of the above-described units 410 to 430 shown in Fig. 2 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the learning device 400 executes a learning data acquisition step performed by the learning data acquisition unit 410, a trained model generation step performed by the trained model generation unit 420, and a trained model output step performed by the trained model output unit 430.

[0034] (Details of the Functional Configuration of the Intelligent Tool 100 According to the First Embodiment) The tool state monitoring unit 133 monitors the output operation of the power unit 120 to monitor the operating state of the intelligent tool 100. For example, if the power unit 120 is configured as an electric screwdriver, the tool state monitoring unit 133 can detect the operating state from the start to the completion of screw tightening by the power unit 120 by monitoring the state of the power unit 120 from the start to the completion of the operation. The tool state monitoring unit 133 also acquires tool-related data related to the intelligent tool 100 by acquiring data output from the power supply unit 110, the power unit 120, and the camera 130. Here, the tool-related data is, for example, data indicating the torque value and torque waveform during screw tightening acquired by a sensor (not shown) of the power unit 120.

[0035] 4A , the image data acquisition unit 131 acquires image data obtained by an imaging sensor (not shown) of the camera 130 and date and time data indicating the date and time when the image data was acquired. Note that, when the tool state monitoring unit 133 detects that the operation of the power unit 120 has been completed, it causes the image data acquisition unit 131 to acquire image data. Therefore, the image data acquired by the image data acquisition unit 131 is image data of the work position and its surroundings when the operation of the power unit 120 has been completed.

[0036] The image data acquisition unit 131 acquires image data for each work position where work is performed with the intelligent tool 100. Therefore, the image data includes, for example, first image data obtained by capturing an image toward a first work position where a first work is performed, and second image data obtained by capturing an image toward a second work position where a second work different from the first work is performed. The image data acquisition unit 131 also acquires multiple types of image data captured from different angles for each work position. That is, the image data includes, for example, multiple types of first image data and multiple types of second image data.

[0037] The data transmission unit 132 transmits the image data and date and time data acquired by the image data acquisition unit 131 to the data collection device 200. In addition, the data transmission unit 132 transmits the tool-related data acquired by the tool state monitoring unit 133 to the data collection device 200.

[0038] (Details of the Functional Configuration of the Data Collecting Device 200 According to the First Embodiment) The data receiving unit 210 receives various types of data, such as the image data, date and time data, and tool-related data, from the intelligent tool 100. The data collecting device 200 stores the various types of data received by the data receiving unit 210 in the data storage unit 220.

[0039] The data transmission unit 230 transmits image data stored in the data storage unit 220 to the personal computer 300. For example, the data transmission unit 230 accesses an engineering environment realized by software installed on the personal computer 300. The data transmission unit 230 then transmits the image data to the personal computer 300 by uploading the image data stored in the data storage unit 220 in the accessed engineering environment as data to be used as learning data, which will be described later.

[0040] (Details of the Functional Configuration of the Personal Computer 300 Pertaining to Embodiment 1) The data receiving unit 310 receives image data from the data collecting device 200, and the learning data generating unit 320 generates learning data including the image data received by the data receiving unit 310. The learning data generating unit 320 generates data as learning data by linking work position identification data that can identify a work position with image data of the work position. The learning data generating unit 320 also performs image processing on the image data, such as changing the brightness, rotating, enlarging, and cropping the image. The learning data generating unit 320 then links the processed image data, together with the original image data, with the work position identification data, to generate learning data.

[0041] For example, as shown in FIG. 4B , the learning data generation unit 320 generates a folder for each work position as work position identification data, and includes image data for the corresponding work position in the folder to generate learning data linking the work position identification data with image data. For example, the learning data generation unit 320 generates learning data including: first learning data linking first work position identification data, which is work position identification data indicating a first work position, with m types of first image data; second learning data linking second work position identification data, which is work position identification data indicating a second work position, with m types of second image data; ..., nth learning data linking nth work position identification data, which is work position identification data indicating an nth work position, with m types of nth image data. In this case, the m types of first image data include first image data after the above-described image processing, and the m types of second image data also include second image data after the above-described image processing. Furthermore, the m types of nth image data also include nth image data after the above-described image processing.

[0042] The learning data generation unit 320 may determine that the image data are of the same work position if the difference between the dates and times the image data were captured is less than a predetermined threshold, for example, less than one minute, based on the date and time data for each acquired image data. By classifying the image data by the date and time of capture in this manner, the learning data generation unit 320 simplifies the process of linking the work position identification data with the image data, making it easier to generate learning data. The learning data generation unit 320 may also classify the image data based on, for example, the similarity of the images. In this case, the data collection device 200 may generate a new trained model capable of calculating the similarity of images by machine learning, such as deep learning, or may separately acquire the new trained model from the learning device 400. In this case, the data collection device 200 may need to be able to calculate the similarity of images by itself, for example, using a method such as feature point extraction, without acquiring a new trained model.

[0043] The setting data generation unit 330 generates setting data used by the data collection device 200. The setting data includes data that enables the data collection device 200 to identify data to be linked to inference result data (described later) and transmitted to the external device 600 from among various data stored in the data storage unit 220. Here, the external device 600 is, for example, a relational database that stores data transmitted by the data collection device 200 for data analysis. The setting data also includes data that identifies the external device 600. The setting data also includes data indicating a format, such as a file format or message format, of data to be transmitted to the external device 600, such as extensible markup language (XML) or JavaScript object notification (JSON). The setting data also includes data indicating a communication protocol, such as transmission control protocol (TCP) or message queueing telemetry transport (MQTT), to be used when transmitting data to the external device 600.

[0044] The data transmitting unit 340 transmits the learning data to the learning device 400. The data transmitting unit 340 also transmits the setting data to the data collecting device 200.

[0045] (Details of the functional configuration of the learning device 400 according to embodiment 1) The learning data acquisition unit 410 acquires learning data from the personal computer 300, and the trained model generation unit 420 generates a trained model for inferring the work position from new image data by machine learning using the learning data acquired by the learning data acquisition unit 410.

[0046] The trained model generation unit 420 learns the relationship between image data and work positions based on training data created based on a combination of image data and work position identification data linked to the image data. In other words, it generates a trained model that infers the most likely work position from new image data.

[0047] The learning algorithm used by the trained model generation unit 420 can be any known algorithm, such as supervised learning, unsupervised learning, or reinforcement learning. In this embodiment, a case where a neural network is applied will be described as an example of the learning algorithm. The trained model generation unit 420 learns the relationship between image data and work positions, for example, by so-called supervised learning in accordance with a neural network model. Here, supervised learning refers to a technique in which a learning device is provided with pairs of input and label data, which are results, to learn the features of the learning data and infer a result from the input.

[0048] A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0049] For example, in a three-layer neural network as shown in Figure 5, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the intermediate layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result changes depending on the values ​​of weights W1 and W2.

[0050] In this embodiment, the neural network learns the relationship between image data and work positions through so-called supervised learning in accordance with learning data created based on a combination of image data acquired by the learning data acquisition unit 410 and the work position identification data to which the image data is linked.

[0051] That is, the neural network learns by inputting image data to the input layer and adjusting the weights W1 and W2 so that the result output from the output layer approaches a work position that can be identified by the work position identification data.

[0052] The trained model generation unit 420 generates and outputs a trained model by performing the above-described learning. In this way, the trained model generation unit 420 can generate a trained model that can classify the first image data and the second image data included in the training data using a neural network, which is an example of classification learning as machine learning. Furthermore, the trained model generation unit 420 can set the number of types of work position identification data included in the training data to a predetermined number of categories and perform classification learning for each image data included in the training data.

[0053] The trained model output unit 430 outputs the trained model by transmitting the trained model to the personal computer 300. In the personal computer 300, the data receiving unit 310 receives the trained model from the learning device 400, and the data transmitting unit 340 transmits the trained model to the data collecting device 200.

[0054] (Details of the functional configuration of the data collection device 200 as an inference device) The data receiving unit 210, which is an example of a trained model acquisition unit, acquires the trained model by receiving the trained model from the personal computer 300. The data collection device 200 stores the trained model in the data storage unit 220, which is an example of a trained model storage unit. The data receiving unit 210, which is an example of an inference data acquisition unit, receives new image data from the intelligent tool 100 and acquires the new image data as inference data.

[0055] In the data collection device 200, the inference result data generation unit 240 inputs the inference data into the trained model to generate inference result data capable of identifying a work position, and stores the generated inference result data in the data storage unit 220. Here, the inference result data is data including score data indicating the score value for each type of work position calculated by the trained model. Therefore, the inference result data is data indicating that the work position with the maximum score value is most likely to be the work position indicated by the new image data. Note that the inference result data is data indicating that the work position indicated by the new image data cannot be identified if the maximum score value does not exceed a predetermined threshold.

[0056] 6, the inference result data includes score data indicating that the score of the first work position is "91," the score of the second work position is "2," ..., and the score of the nth work position is "1." In this case, for example, if the predetermined threshold is "40," the inference result data indicates that the first work position with the maximum score value is most likely to be the work position indicated by the new image data.

[0057] Furthermore, in the data collection device 200, the data receiving unit 210 receives setting data from the personal computer 300, and also receives data necessary for production management from a production management device 500 that is provided outside the work management system 1 and manages the production of workpieces. Here, the data necessary for production management includes, for example, worker identification data that can identify the worker, such as the worker's employee number, and workpiece identification data that can identify the workpiece, such as the workpiece's serial number. The data collection device 200 also stores these various types of data received by the data receiving unit 210 in the data storage unit 220.

[0058] Furthermore, based on the setting data, the data collection device 200 identifies data to be linked to the inference result data and transmitted to the external device 600 from among various types of data such as worker identification data, inference data, date and time data, tool-related data, and workpiece identification data. The data transmission unit 230 then transmits the inference result data in a linked state with the identified data.

[0059] (Regarding Trained Model Generation Process According to Embodiment 1) Next, an operation of the learning device 400 to generate and output a trained model will be described using a flowchart. When the learning device 400 is powered on, it starts executing the trained model generation process shown in FIG. 7. First, the learning data acquisition unit 410 acquires learning data from the personal computer 300 (step S101). For example, the learning data acquisition unit 410 acquires, as the learning data, first learning data in which first work position identification data, which is work position identification data indicating the first work position shown in FIG. 4B, is linked to multiple types of first image data; second learning data in which second work position identification data, which is work position identification data indicating the second work position, is linked to multiple types of second image data; ..., nth learning data in which nth work position identification data, which is work position identification data indicating the nth work position, is linked to multiple types of nth image data.

[0060] Next, the trained model generation unit 420 generates a trained model for inferring a work position from new image data through machine learning using the acquired training data (step S102). Specifically, the trained model generation unit 420 inputs image data included in the training data into the input layer shown in FIG. 5 and generates, as a trained model, a neural network in which the weights W1 and W2 are adjusted so that the result output from the output layer approaches a work position identifiable by the work position identification data linked to the image data. The trained model output unit 430 then outputs the trained model by transmitting it to the personal computer 300 (step S103), and the process ends.

[0061] (Regarding the inference result data generation process according to embodiment 1) Next, using a flowchart, an operation of the data collection device 200, as an example of an inference device, to generate and output inference result data will be described. When the data collection device 200 is powered on, it starts executing the inference result data generation process shown in Fig. 8. First, the data receiving unit 210, as an example of a trained model acquisition unit, acquires the trained model by receiving it from the personal computer 300 (step S201).

[0062] Next, the data receiving unit 210, which is an example of an inference data acquiring unit, receives new image data from the intelligent tool 100 and acquires the new image data as inference data (step S202). Next, the inference result data generating unit 240 inputs the inference data into the learned model to generate inference result data, which is data including score data indicating a score value for each type of work position (step S203). Then, the data collecting device 200 identifies data to be linked to the inference result data and transmitted to the external device 600 based on the setting data (step S204). The data transmitting unit 230 transmits the inference result data linked to the identified data, thereby outputting the inference result data (step S205), and the process ends.

[0063] As described above, according to the work management system 1 of the present embodiment, in the intelligent tool 100, the camera 130 captures an image of a work position where work is being performed on a workpiece using the intelligent tool 100. In the learning device 400, the learning data acquisition unit 410 acquires learning data including image data previously captured by the camera 130, and the trained model generation unit 420 generates a trained model for inferring a work position from new image data through machine learning using the learning data. In the data collection device 200 as an example of an inference device, the data receiving unit 210 as an example of an inference data acquisition unit acquires new image data captured by the camera 130 as inference data. In the data collection device 200, the inference result data generation unit 240 inputs the inference data into the trained model to generate inference result data that can identify the work position indicated by the new image data.

[0064] By doing this, the work management system 1 according to this embodiment can identify the work position indicated by the new image data and can manage work on the workpiece as the object to be processed using the intelligent tool 100.

[0065] Here, there are various methods for image recognition processing using machine learning, broadly including object detection, classification, segmentation, and anomaly detection. Of these methods, it is natural for those skilled in the art to select an object recognition method in fields such as identifying the work position relative to a workpiece and the work target, as in the present embodiment. For example, the system disclosed in Patent Document 2 employs an object recognition method that recognizes objects in captured images through pattern recognition processing. However, as described above, the object recognition method employed by the system disclosed in Patent Document 2 has the problem that, for example, if the resolution of the captured image is low, it is difficult to recognize the objects in the captured image.

[0066] In contrast, in the work management system 1 according to the present embodiment, the learning data includes first image data obtained by imaging toward the first work position where the first work is being performed, second image data obtained by imaging toward the second work position where the second work is being performed, ..., nth image data obtained by imaging toward the nth work position where the nth work is being performed. The trained model generation unit 420 then employs classification learning as machine learning, i.e., a classification method, to generate a trained model capable of classifying each image data from data obtained from around the work position.

[0067] By doing so, in the work management system 1 according to the present embodiment, the trained model generation unit 420 learns to infer the work position based on data obtained from the periphery of the work position rather than data obtained from the work position itself, rather than recognizing objects at the work position. It is clear that the amount of information in the data obtained from the periphery of the work position is far greater than the amount of information in the data obtained from the work position itself. Therefore, the work management system 1 according to the present embodiment can generate inference result data that can identify the work position as one of the work positions, even when the image data has a low resolution, compared to the system disclosed in Patent Document 2. As a result, the work management system 1 according to the present embodiment has a higher recognition rate for work performed on a workpiece using the intelligent tool 100 than a work management system that does not generate a trained model capable of classifying each image data from data obtained from the periphery of the work position through classification learning.

[0068] Furthermore, according to the work management system 1 of this embodiment, the trained model generation unit 420 of the learning device 400 performs classification learning for each image data included in the learning data. In this way, the work management system 1 of this embodiment can identify which of the work positions the work position indicated by new image data is, simply by learning the image data captured by the camera 130.

[0069] Furthermore, according to the work management system 1 of the present embodiment, in the learning device 400, the trained model generation unit 420 performs classification learning based on the number of types of work position identification data linked by the personal computer 300, i.e., the number of categories. In this way, the work management system 1 of the present embodiment can perform classification learning more efficiently than a work management system that does not perform classification learning based on a predetermined number of categories. As a result, the work management system 1 of the present embodiment improves the accuracy of identifying which of the work positions the work position indicated by new image data is, and further improves the recognition rate of work on a workpiece using the intelligent tool 100.

[0070] Furthermore, according to the work management system 1 of this embodiment, the learning data generation unit 320 in the personal computer 300 generates learning data by linking the processed image data, which has been subjected to image processing, together with the original image data and the work position identification data. Therefore, in the learning device 400, the learning data acquisition unit 410 acquires learning data that includes not only the original image data but also the processed image data.

[0071] In this way, the processed image data can be image data resulting from a simulation, that is, a reproduction, through image processing, of changes in an image according to the situation in which the camera 130 captures an image. Thus, the trained model generation unit 420 can generate a trained model as a result of simulating various changes in the situation in which the camera 130 captures an image, by machine learning the training data including the processed image data. As a result, the work management system 1 according to the present embodiment has improved accuracy in identifying which of the work positions the work position indicated by the new image data is, compared to a work management system that does not acquire training data including the processed image data, and further improves the recognition rate of work performed on a workpiece using the intelligent tool 100.

[0072] Furthermore, according to the work management system 1 of this embodiment, the data collection device 200 identifies data to be linked to the inference result data from various types of data, such as worker identification data, inference data, date and time data, tool-related data, and workpiece identification data, based on the setting data acquired from the personal computer 300. The data transmission unit 230 then transmits the inference result data and the identified data in a linked state to the external device 600.

[0073] In this way, the external device 600 can perform data analysis using the work location identified from the inference result data and the work-related data linked to the inference result data. In particular, if the external device 600 is configured as a relational database, when inserting or updating the work location identified from the inference result data, the linked data can be used as identification information, making it possible to manage the acquired data more effectively than when the inference result data and various data are not linked.

[0074] Furthermore, according to the work management system 1 of this embodiment, the camera 130 is provided in the intelligent tool 100 and is supplied with power from the power supply unit 110, and the data transmission unit 132 of the camera 130 transmits the image data obtained by capturing the image to the data collection device 200 via wireless communication.

[0075] In this manner, the power supply unit 110 of the intelligent tool 100 secures power for the camera 130, which consumes significantly less power than the power unit 120, so that the intelligent tool 100 can realize the functions of the units 131 to 133 of the camera 130 with almost no loss in power supply to the power unit 120. In addition, in this manner, the data collection device 200 can acquire data from the intelligent tool 100 via wireless communication. As a result, the work management system 1 according to this embodiment improves the efficiency of work by the intelligent tool 100 compared to a work management system that does not collect data from the intelligent tool via wireless communication.

[0076] Furthermore, according to the work management system 1 of this embodiment, the inference result data is data including score data indicating the score value for each type of work position calculated by the trained model. The inference result data indicates that the work position having the largest score value among the score values ​​exceeding a predetermined threshold is most likely to be the work position indicated by the new image data. In this way, the work management system 1 of this embodiment has improved accuracy in identifying which of the work positions is the work position indicated by the new image data compared to a work management system whose inference result data does not include score data, and further improves the recognition rate of work on a workpiece using the intelligent tool 100.

[0077] [Embodiment 2] In the first embodiment, the data collection device 200 generates and outputs inference result data as an inference device, but this is not limited to this. For example, the intelligent tool 100 may generate and output inference result data as an inference device. Below, with reference to FIG. 9, the work management system 1 according to the second embodiment will be described in detail. Note that in the second embodiment, only the configuration different from the first embodiment will be described, and the description of the same configuration as the first embodiment will be omitted to avoid redundancy.

[0078] 9 , in the work management system 1 according to the present embodiment, the camera 130 of the intelligent tool 100, which is an example of an inference device, includes an inference result data generation unit 240, while the inference result data generation unit 240 is omitted from the data collection device 200. In addition, a data receiving unit 134 that receives data is added to the camera 130, and an imaging trigger data generation unit 250 that generates imaging trigger data, which will be described later, is added to the data collection device 200.

[0079] (Hardware Configuration of the Intelligent Tool 100 According to the Second Embodiment) In the intelligent tool 100, the control unit 51 shown in Fig. 3 functions as the inference result data generation unit 240 shown in Fig. 9 in accordance with a control program 59, and the transmission / reception unit 56 shown in Fig. 3 functions as the data receiving unit 134 shown in Fig. 9. Returning to Fig. 3, the intelligent tool 100 realizes the function of the inference result data generation unit 240 shown in Fig. 9 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes the inference result data generation step performed by the inference result data generation unit 240.

[0080] (Hardware Configuration of Data Collection Device 200 According to Embodiment 2) In the data collection device 200, the control unit 51 shown in Fig. 3 functions as the imaging trigger data generation unit 250 shown in Fig. 9 in accordance with a control program 59. Returning to Fig. 3, in the data collection device 200, the control unit 51 uses the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources, thereby realizing the function of the imaging trigger data generation unit 250 shown in Fig. 9. For example, the data collection device 200 executes the imaging trigger data generation step performed by the imaging trigger data generation unit 250.

[0081] (Details of the Functional Configuration of the Data Collection Device 200 According to the Second Embodiment) The imaging trigger data generation unit 250 generates imaging trigger data that instructs the camera 130 to capture an image, based on the tool-related data received by the data receiving unit 210. For example, the imaging trigger data generation unit 250 generates imaging trigger data when it is determined from the tool-related data that the operation of the power unit 120 has been completed. The imaging trigger data generation unit 250 may also generate imaging trigger data based on the tool-related data and setting data. In this case, in the personal computer 300, the setting data generation unit 330 needs to generate setting data that includes data related to the generation of imaging trigger data, such as data indicating the conditions for generating imaging trigger data from the tool-related data and data indicating the timing for generating and transmitting the imaging trigger data.

[0082] The data transmission unit 230 transmits the imaging trigger data generated by the imaging trigger data generation unit 250 to the intelligent tool 100. In addition, the data transmission unit 230 transmits the trained model to the intelligent tool 100.

[0083] (Details of the functional configuration of the intelligent tool 100 as an example of an inference device) The data receiving unit 134 as an example of a trained model acquisition unit acquires the trained model by receiving the trained model from the data collecting device 200. The data receiving unit 134 also acquires imaging trigger data from the data collecting device 200. The image data acquiring unit 131 as an example of an inference data acquisition unit acquires new image data when the data receiving unit 134 receives the imaging trigger data, thereby acquiring inference data.

[0084] The inference result data generation unit 240 inputs the inference data into the learned model to generate inference result data that can identify the work position. The data transmission unit 132 transmits the inference data, which is new image data acquired by the image data acquisition unit 131, and date and time data to the data collection device 200. The data transmission unit 132 also transmits the inference result data generated by the inference result data generation unit 240 to the data collection device 200.

[0085] As described above, according to the work management system 1 of the present embodiment, the camera 130 of the intelligent tool 100 includes the inference result data generation unit 240. In this manner, in the work management system 1 of the present embodiment, the camera 130 of the intelligent tool 100 can generate and output inference result data as an inference device. That is, in the present embodiment, unlike in the first embodiment, the intelligent tool 100 can generate and output inference result data as an inference device itself, without transmitting new image data as inference data to the data collection device 200 as an inference device. As a result, in the work management system 1 of the present embodiment, the intelligent tool 100 can quickly perform inference as an inference device using new image data that it has acquired itself, without having to increase the communication speed of the network between the intelligent tool 100 and the inference device so that new image data can be quickly acquired.

[0086] Furthermore, according to the work management system 1 of this embodiment, the camera 130 is provided in the intelligent tool 100 and is supplied with power from the power supply unit 110, and the data transmission unit 132 of the camera 130 transmits the inference result data generated by the inference result data generation unit 240 to the data collection device 200 by wireless communication. In this manner, the power for the camera 130, which consumes significantly less power than the power unit 120, is secured from the power supply unit 110 of the intelligent tool 100, so that the intelligent tool 100 can realize the functions of the respective units 131 to 134 and 240 of the camera 130 with almost no loss in the power supply to the power unit 120.

[0087] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, the imaging trigger data generation unit 250 generates imaging trigger data based on the tool-related data received from the intelligent tool 100, and the data transmission unit 230 transmits the imaging trigger data to the intelligent tool 100. Then, in the intelligent tool 100, when the camera 130 receives the imaging trigger data from the data collection device 200, the image data acquisition unit 131, which is an example of an inference data acquisition unit, acquires new image data as inference data.

[0088] In this way, the work management system 1 according to the present embodiment can instruct the timing of acquisition of inference data and inference by the intelligent tool 100 from the data collection device 200, which can transmit and receive data to and from the intelligent tool 100. In addition, the work management system 1 according to the present embodiment has the same effects as the work management system 1 according to the first embodiment.

[0089] [Third Embodiment] In the first and second embodiments, the data collecting device 200 and the intelligent tool 100 function as inference devices and output the inference result data to the external device 600, and the inference result data is used by the external device 600. However, this is not limited to this, and the inference result data may be used within the work management system 1. Below, the work management system 1 according to the third embodiment will be described in detail with reference to Figures 10 to 12. Note that in the third embodiment, configurations different from those in the first embodiment will be described, and descriptions of the same configurations as those in the first embodiment will be omitted to avoid redundancy.

[0090] (Regarding the intelligent tool 100 according to the third embodiment) As shown in FIG. 10, in the intelligent tool 100, the camera 130 further includes a data receiving unit 134 similar to that in the second embodiment, and a tool operation control unit 135 that controls the operation of the tool.

[0091] (Regarding the data collection device 200 according to the third embodiment) The data collection device 200 further includes a work order determination unit 260 that determines the order of work, an activation data generation unit 270 that generates activation data that activates the operation of the intelligent tool 100, an abnormal order data generation unit 280 that generates abnormal order data that indicates an abnormality in the order of work, and an abnormal order data output unit 290 that outputs the abnormal order data.

[0092] (Hardware Configuration of Intelligent Tool 100 According to Embodiment 3) In the intelligent tool 100, the control unit 51 shown in Fig. 3 functions as a tool operation control unit 135 shown in Fig. 10 in accordance with a control program 59, and the transmission / reception unit 56 shown in Fig. 3 functions as a data receiving unit 134 shown in Fig. 10. Returning to Fig. 3, in the intelligent tool 100, the control unit 51 realizes the function of the tool operation control unit 135 shown in Fig. 10 by using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the intelligent tool 100 executes the tool operation control steps performed by the tool operation control unit 135.

[0093] (Hardware Configuration of Data Collection Apparatus 200 Pertaining to Embodiment 3) In addition, in the data collection apparatus 200, the control unit 51 shown in FIG. 3 functions as an operation order determination unit 260, an enabling data generation unit 270, and an out-of-order data generation unit 280 shown in FIG. 10 in accordance with a control program 59, and the display unit 55 shown in FIG. 3 functions as an out-of-order data output unit 290 shown in FIG. 10. Returning to FIG. 3, the data collection apparatus 200 realizes the functions of the units 260 to 290 shown in FIG. 10 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the data collection apparatus 200 executes an operation order determination step performed by the operation order determination unit 260 and an enabling data generation step performed by the enabling data generation unit 270. In addition, for example, the data collection apparatus 200 executes an out-of-order data generation step performed by the out-of-order data generation unit 280 and an out-of-order data output step performed by the out-of-order data output unit 290.

[0094] (Details of the functional configuration of the data collection device 200 according to embodiment 3) The inference result data generation unit 240 generates inference result data each time the data receiving unit 210, which is an example of an inference data acquisition unit, acquires inference data, and stores the generated inference result data in the data storage unit 220. The data receiving unit 210 receives order data of work positions that is predetermined based on the order of work on the workpieces, as data necessary for manufacturing management, from the manufacturing management equipment 500. The data collection device 200 stores the order data received by the data receiving unit 210 in the data storage unit 220.

[0095] The work order determination unit 260 determines whether the order of work on the workpiece is correct by comparing the history data of the work position identified from the inference result data stored in the data storage unit 220 with the order data stored in the data storage unit 220. When the work order determination unit 260 determines that the order of work on the workpiece is correct, the activation data generation unit 270 generates activation data that activates the operation of the intelligent tool 100 to be used for the next work on the workpiece. When the activation data generation unit 270 generates the activation data, the data transmission unit 230 transmits the activation data to the intelligent tool 100 to be used for the next work.

[0096] The abnormal sequence data generating unit 280 generates abnormal sequence data when the work order determining unit 260 determines that the order of work on the workpieces is incorrect. The abnormal sequence data output unit 290 outputs the abnormal sequence data by, for example, displaying an abnormal sequence image shown in FIG. 11 on the display unit 55 or outputting an alarm sound.

[0097] (Details of the Functional Configuration of the Intelligent Tool 100 According to the Third Embodiment) The data receiving unit 134 receives activation data from the data collecting device 200. The tool operation control unit 135 disables the output operation of the power unit 120 until the data receiving unit 134 receives the activation data, and controls the output operation of the power unit 120 to be activated when the data receiving unit 134 receives the activation data.

[0098] (Work Order Determination Process According to Embodiment 3) Next, the operation of the data collection device 200 to determine the work order and transmit activation data will be described using a flowchart. When the inference result data generation unit 240 generates inference result data and stores it in the data storage unit 220, the data collection device 200 starts executing the work order determination process shown in Fig. 12. First, the work order determination unit 260 identifies history data of the work location from multiple types of inference result data stored in the data storage unit 220 (step S301), and acquires the order data stored in the data storage unit 220 (step S302).

[0099] Next, the work order determination unit 260 compares the history data with the order data to determine whether the order of the work on the workpiece is correct (step S303). If the work order determination unit 260 determines that the work order is correct (step S303; Y), the validation data generation unit 270 generates validation data (step S304). The data transmission unit 230 transmits the validation data to the intelligent tool 100 to be used for the next work (step S305), and the process ends. On the other hand, if the work order determination unit 260 determines that the work order is incorrect (step S303; N), the sequence abnormality data generation unit 280 generates sequence abnormality data (step S306). The sequence abnormality data output unit 290 outputs the sequence abnormality data by, for example, displaying the sequence abnormality image shown in FIG. 11 on the display unit 55 or outputting an alarm (step S307), and the process ends.

[0100] As described above, according to the work management system 1 of this embodiment, in the data collection device 200, the inference result data generation unit 240 generates inference result data each time the data receiving unit 210 (an example of an inference data acquisition unit) acquires inference data, and stores the generated inference result data in the data storage unit 220. The work order determination unit 260 identifies work position history data from the multiple types of inference result data stored in the data storage unit 220. The work order determination unit 260 then compares the identified work position history data with predetermined work position order data based on the order of work tasks for the work stored in the data storage unit 220, thereby determining whether the order of work tasks for the work is correct. In this way, the work management system 1 of this embodiment can use the inference result data to determine whether the order of work tasks for the work is correct.

[0101] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, when the work order determination unit 260 determines that the order of work on the workpiece is correct, the activation data generation unit 270 generates activation data, and the data transmission unit 230 transmits the activation data to the intelligent tool 100 to be used for the next work. Then, in the intelligent tool 100, the tool operation control unit 135 controls the power unit 120 to activate the output operation when the data receiving unit 134 receives the activation data. In this way, the work management system 1 of this embodiment can prevent work that has been carried out in an incorrect order from being continued.

[0102] Furthermore, according to the work management system 1 of this embodiment, in the data collection device 200, when the work order determination unit 260 determines that the order of work on the work is incorrect, the sequence abnormality data generation unit 280 generates sequence abnormality data, and the sequence abnormality data output unit 290 outputs the sequence abnormality data. In this way, the work management system 1 of this embodiment can notify system stakeholders such as workers and users that work has been performed in an incorrect order. In addition, the work management system 1 of this embodiment achieves the same effects as the work management system 1 of embodiment 1.

[0103] [Modification] In the above-described first to third embodiments, a worker performs work on a workpiece using the intelligent tool 100, but this is not limited to this. For example, a robot may perform work on a workpiece using the intelligent tool 100.

[0104] As in the above-mentioned embodiments 1 to 3, it is preferable to use the learning device 400 as a cloud server for machine learning, but this is not limited to this. For example, the data collection device 200 and the personal computer 300 may include the functions of each of the parts 410 to 430 of the learning device 400, and these computer devices may operate as learning devices.

[0105] In the first to third embodiments, the functions of the inference device are included in the intelligent tool 100 or the data collection device 200, and the intelligent tool 100 or the data collection device 200 operates as the inference device, but this is not limiting. For example, the functions of the inference device may be included in the personal computer 300 or the learning device 400, and the personal computer 300 or the learning device 400 may operate as the inference device. If the learning device 400 also operates as the inference device, the inference by the inference device will also be implemented as a cloud system.

[0106] In the above-described first to third embodiments, the case where supervised learning is applied to the learning algorithm used by the trained model generation unit 420 has been described, but this is not limited to this. As for the learning algorithm, reinforcement learning, unsupervised learning, semi-supervised learning, or the like can also be applied in addition to supervised learning.

[0107] The trained model generation unit 420 may also learn the work position indicated by new image data according to training data created for the same work at multiple types of work sites. It is also possible to add or remove work positions from which training data is collected during the process. Furthermore, the learning device 400 that has trained to infer the work position indicated by new image data for a certain work may be applied to a different work, and the work position indicated by the new image data for the different work may be re-trained and updated.

[0108] In addition, the learning algorithm used in the trained model generation unit 420 can be deep learning, which learns to extract the features themselves, or machine learning can be performed according to other known methods, such as genetic programming, functional logic programming, support vector machines, etc.

[0109] In the above-described first to third embodiments, the work position indicated by new image data is inferred using a trained model trained on work performed at a work site managed by the work management system 1. However, this is not limiting. For example, a trained model generated by machine learning in an external system on the same work performed at another work site not managed by the work management system 1 may be obtained, and the acquired trained model may be used to infer the work position indicated by new image data.

[0110] In the first to third embodiments, the intelligent tool 100 and the data collecting device 200 are connected via a wireless LAN. However, this is not a limitation. For example, data may be transmitted and received via wireless communication using another wireless communication standard, such as Bluetooth (registered trademark). In this case, at least the intelligent tool 100 and the data collecting device 200 must be paired to establish a state in which they can communicate one-to-one. Furthermore, as in the first to third embodiments, it is preferable that the intelligent tool 100 and the data collecting device 200 can transmit and receive data via wireless communication. However, this is not a limitation. For example, the intelligent tool 100 and the data collecting device 200 may transmit and receive data via wired communication, such as a wired LAN.

[0111] As in the above-described embodiments 1 to 3, it is preferable that the camera 130 is provided with an illumination light output device to increase the brightness of the captured image, but this is not limited to this as long as the image can be captured by the imaging sensor, and the illumination light output device may be omitted from the camera 130.

[0112] In the above-described first and third embodiments, the camera 130 automatically acquires image data when it detects that the power unit 120 has completed its operation, and in the above-described second embodiment, the camera 130 acquires image data when it receives imaging trigger data, but the conditions for acquiring image data are not limited to these. For example, the camera 130 may acquire image data when an operator operates an imaging switch provided on the camera 130. In other words, the camera 130 may acquire image data manually by the operator operating the imaging switch.

[0113] As in the above-described first to third embodiments, it is preferable that the learning data generation unit 320 generates a folder for each work position as work position identification data and includes image data of the corresponding work position in the folder, but this is not limited to this. For example, the learning data may be generated by linking the work position identification data and the image data by adding the work position identification data to the image data.

[0114] In the first and third embodiments, the camera 130 acquires image data when it detects the completion of the operation of the power unit 120. However, this is not limited to this. Multiple types of image data may be acquired from the start of operation of the power unit 120 to the completion of operation. In this case, the inference data may be multiple types of new image data, and the inference device may input all image data as inference data into the trained model and identify the work position with the highest score value as the work position indicated by the inference data. Furthermore, in this case, the inference data may be the image data that is most suitable for inference from the multiple types of new image data. By doing this, even if the acquired image data includes low contrast and sharpness due to the time required for various adjustments such as autofocus and automatic exposure adjustment, the image data that is statistically considered to have the best contrast and sharpness can be used as the inference data.

[0115] In the first to third embodiments, the inference device infers that the work position indicated by the new image data cannot be identified if the maximum score value indicated by the score data included in the inference result data does not exceed a threshold value. However, this is not limited to this. For example, the inference device may omit the process of determining whether the maximum score value exceeds a threshold value, and infer that the work position with the maximum score value is the work position indicated by the new image data.

[0116] In the above-described first to third embodiments, the data collection device 200 updates various data that may be linked to inference result data based on the setting data to the latest data when the intelligent tool 100 and the manufacturing control device 500 transmit the data. However, this is not limited to this. For example, the data collection device 200 may request the intelligent tool 100 and the manufacturing control device 500 to transmit various data at a predetermined polling interval, thereby updating the various data to the latest data at the polling interval. In this case, for example, the personal computer 300 may generate setting data including collection condition data indicating conditions, such as the polling interval, for causing the data collection device 200 to collect the various data described above, and transmit the setting data to the data collection device 200 to set conditions for the data collection device 200 to update the various data to the latest data.

[0117] In the above-described first to third embodiments, the data collection device 200 transmits the inference result data and various data acquired from the intelligent tool 100 or the manufacturing control device 500 in a linked state to the external device 600. However, the present invention is not limited to this, and the data does not have to be transmitted to the external device 600. In this case, the data collection device 200 may store the data in which the inference result data and various data are linked in the data storage unit 220 in a data format suitable for data analysis, such as CSV (Comma Separated Value). In this way, the data collection device 200, like the external device 600, can perform data analysis using the work position identified from the inference result data and the work-related data linked to the inference result data.

[0118] In the third embodiment, when the data collection device 200 determines that the order of operations on the workpiece is correct, it generates validation data and transmits it to the intelligent tool 100 used for the next operation. However, the processing performed when it is determined that the order of operations is correct is not limited to this. For example, when the data collection device 200 determines that the order of operations on the workpiece is correct, it may generate data indicating that the order of operations was correct and transmit the data to the intelligent tool 100 that performed the operation. In this case, the data collection device 200 may determine whether the operation completion state is appropriate based on the tool-related data, and if not, transmit data indicating that to the intelligent tool 100. For example, when it determines that a screw is not sufficiently tightened based on data indicating a torque value or data indicating a torque waveform, it may transmit data indicating that to the intelligent tool 100. In this way, the intelligent tool 100 can recognize the information and allow the worker to appropriately perform the operation at the inferred operation position.

[0119] In the third embodiment, the data collection device 200 outputs the sequence anomaly data by displaying the sequence anomaly image shown in FIG. 11 on the display unit 55 or by outputting an alarm sound. However, the output of the sequence anomaly data is not limited to this. For example, the sequence anomaly data may be transmitted to another device, such as the intelligent tool 100 that performed the work, the intelligent tool 100 that will perform the next work, the manufacturing management device 500, or the external device 600, i.e., output externally. In this way, the user of the other device that receives the sequence anomaly data can recognize that the work was performed in an incorrect order.

[0120] The core processing parts of the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400, each of which includes the control unit 51, the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, the transmission / reception unit 56, and the internal bus 50, can be realized using a general computer system rather than a dedicated system. For example, a computer program for executing the above operations may be stored and distributed on a computer-readable recording medium, such as a flexible disk or a DVD-ROM (Read-Only Memory), and the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400 may be configured to execute the above processes by installing the computer program on a computer. Alternatively, the computer program may be stored in a storage device of a server device on a communication network and downloaded by a general computer system to configure the intelligent tool 100, the data collecting device 200, the personal computer 300, and the learning device 400.

[0121] In addition, when the functions of the intelligent tool 100, the data collection device 200, the personal computer 300, and the learning device 400 are realized by sharing the functions of an OS (Operating System) and an application program, or when the functions are realized by cooperation between an OS and an application program, only the application program portion may be stored in a recording medium or storage device.

[0122] It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and provided via the network. The computer program may then be started and executed under the control of an OS in the same way as other application programs, thereby performing the above-mentioned processing.

[0123] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure.

[0124] 1...work management system, 50...internal bus, 51...control unit, 52...main memory unit, 53...external memory unit, 54...operation unit, 55...display unit, 56...transmission / reception unit, 59...control program, 100...intelligent tool, 101...first intelligent tool, 102...second intelligent tool, 109...ninth intelligent tool, 110...power supply unit, 120...power unit, 130...camera, 131...image data acquisition unit, 132...data transmission unit, 133...tool state monitoring unit, 134...data reception unit, 135...tool operation control unit, 200...data collection device, 210...data reception unit , 220...data storage unit, 230...data transmission unit, 240...inference result data generation unit, 250...imaging trigger data generation unit, 260...work sequence determination unit, 270...validation data generation unit, 280...sequence abnormality data generation unit, 290...sequence abnormality data output unit, 300...personal computer, 310...data receiving unit, 320...learning data generation unit, 330...setting data generation unit, 340...data transmission unit, 400...learning device, 410...learning data acquisition unit, 420...trained model generation unit, 430...trained model output unit, 500...manufacturing management equipment, 600...external device.

Claims

1. A work management system for managing operations on a work piece as a processing object using tools, comprising: a learning data acquisition unit that acquires learning data including image data obtained by imaging in advance by an imaging member that images toward a work position where an operation is being performed on the work piece using the tool; a learned model generation unit that generates a learned model for inferring the work position from new image data by machine learning using the learning data; an inference data acquisition unit that acquires the new image data obtained by imaging by the imaging member as inference data; an inference result data generation unit that, each time the inference data acquisition unit acquires the inference data, inputs the acquired inference data into the learned model to generate inference result data capable of specifying the work position; a work order determination unit that determines whether the order of operations on the work piece is correct by comparing the history data of the work position specified from the plurality of types of inference result data generated by the inference result data generation unit with the order data of the work position predetermined based on the order of operations on the work piece; and comprising: the learning data includes first image data obtained by imaging toward a first work position as the work position where a first operation is being performed on the work piece using a first tool as the tool, and second image data obtained by imaging toward a second work position as the work position where a second operation different from the first operation is being performed on the work piece using a second tool as a tool of a type different from the first tool; the learned model generation unit generates the learned model capable of classifying the first image data and the second image data from data obtained from the periphery of the work position by classification learning as the machine learning; an activation data generation unit that generates activation data for activating the operation of the second tool when the work order determination unit determines that the order of the first operation using the first tool on the work piece is correct and the operation following the first operation on the work piece is the second operation; a data transmission unit that transmits the activation data to the second tool when the activation data generation unit generates the activation data for activating the operation of the second tool; and further comprising: a work management system.

2. The trained model generation unit performs the classification learning for each of the image data included in the learning data. The work management system according to claim 1.

3. The trained model generation unit performs the classification learning based on a predetermined number of classifications. The work management system according to claim 1.

4. The learning data acquisition unit acquires the learning data including the processed image data obtained by performing image processing on the image data previously captured by the imaging member. The work management system according to claim 1.

5. The imaging member includes the inference result data generation unit. The work management system according to claim 1.

6. An operator performs the work. A data transmission unit that transmits to an external device at least any one of the operator identification data that can identify the operator, the date and time data indicating the date and time when the new image data serving as the inference data was acquired, the work identification data that can identify the work, the tool-related data regarding the tool, and the inference result data associated with the one data. further comprising The work management system according to claim 1.

7. A data collection device that collects data related to the management of the work on the work. further comprising Power is supplied to the imaging member from the tool. The imaging member includes a data transmission unit that transmits the image data obtained by imaging to the data collection device by wireless communication. The work management system according to claim 1.

8. A data collection device that collects data related to the management of the work on the work. further comprising Power is supplied to the imaging member from the tool. The imaging member includes a data transmission unit that transmits the inference result data generated by the inference result data generation unit to the data collection device by wireless communication. The work management system according to claim 5.

9. The inference result data includes score data indicating score values for each predetermined classification calculated by the trained model. The work position that can be specified from the inference result data is a position specified by one classification that is the maximum value among the values exceeding a predetermined threshold value of the score value. The work management system according to claim 1.

10. The imaging member performs imaging a plurality of times from the start of a work on the work using the tool until the work is completed. The inference data acquisition unit acquires the inference data based on a plurality of types of the new image data obtained by the imaging member imaging the one operation a plurality of times. The work management system according to claim 1.

11. Learning data including image data obtained by imaging in advance by an imaging unit that images toward a work position where work is being performed on a work as a processing object using a tool, the first image data obtained by imaging toward a first work position as the work position where a first operation is being performed on the work using a first tool as the tool, and a second operation different from the first operation is being performed on the work using a second tool as a tool of a type different from the first tool, the second image data obtained by imaging toward a second work position as the work position, a learning data acquisition unit that acquires the learning data. A learned model for inferring the work position from new image data by machine learning using the learning data, a learned model generation unit that generates the learned model capable of classifying the first image data and the second image data by classification learning as the machine learning. An inference data acquisition unit that acquires the new image data obtained by the imaging unit as inference data. An inference result data generation unit that, each time the inference data acquisition unit acquires the inference data, inputs the acquired inference data into the learned model to generate inference result data capable of specifying the work position. A work order determination unit that determines whether the order of work on the work is correct by comparing the history data of the work position specified from a plurality of types of the inference result data generated by the inference result data generation unit with the order data of the work position predetermined based on the order of work on the work. An activation data generation unit that generates activation data for activating the operation of the second tool when the work order determination unit determines that the order of the first operation using the first tool on the work is correct and the next operation on the work after the first operation is the second operation. A data transmission unit that transmits the activation data to the second tool when the activation data generation unit generates the activation data for activating the operation of the second tool. Comprising Learning device.

12. A learned model for inferring the working position from new image data generated by machine learning using learning data including image data previously captured by an imaging unit that captures an image toward a working position where an operation is being performed on a workpiece as an object to be processed. The learning data includes first image data obtained by imaging toward a first working position that is the working position where a first operation is being performed on the workpiece using a first tool as the tool, and second image data obtained by imaging toward a second working position that is the working position where a second operation different from the first operation is being performed on the workpiece using a second tool that is a different type of tool from the first tool. The learned model is a learned model capable of classifying the first image data and the second image data by taxonomy learning as the machine learning, and a learned model acquisition unit that acquires the learned model, An inference data acquisition unit that acquires the new image data captured by the imaging unit as inference data, An inference result data generation unit that, each time the inference data acquisition unit acquires the inference data, inputs the acquired inference data into the learned model to generate inference result data capable of specifying the working position, An operation order determination unit that determines whether the operation order for the workpiece is correct by comparing the history data of the working position specified from the plurality of types of inference result data generated by the inference result data generation unit with the order data of the working position predetermined based on the order of operations on the workpiece, An activation data generation unit that generates activation data for activating the operation of the second tool when the operation order determination unit determines that the order of the first operation using the first tool on the workpiece is correct and the next operation after the first operation on the workpiece is the second operation, A data transmission unit that transmits the activation data to the second tool when the activation data generation unit generates the activation data for activating the operation of the second tool, Comprising, Inference device.

13. A learning data acquisition step of acquiring learning data including image data preliminarily captured by an imaging unit that captures an image toward a working position where a computer is performing an operation on a workpiece as an object to be processed, the learning data including first image data captured toward a first working position that is the working position where a first operation is being performed on the workpiece using a first tool as the tool, and second image data captured toward a second working position that is the working position where a second operation different from the first operation is being performed on the workpiece using a second tool as a tool of a type different from the first tool. A learned model generation step of generating, by the computer, a learned model for inferring the working position from new image data by machine learning using the learning data, the learned model being capable of classifying the first image data and the second image data by classification learning as the machine learning. A learned model output step of the computer outputting the learned model to an inference device that performs inference using the learned model. An inference data acquisition step of the computer acquiring the new image data captured by the imaging unit as inference data. An inference result data generation step of the computer, each time the inference data is acquired in the inference data acquisition step, inputting the acquired inference data into the learned model to generate inference result data capable of specifying the working position. A working order determination step of the computer comparing history data of the working position specified from the plurality of types of inference result data generated in the inference result data generation step with order data of the working position predetermined based on the order of operations on the workpiece, to determine whether the order of operations on the workpiece is correct. An activation data generation step of the computer generating activation data for activating the operation of the second tool when it is determined in the working order determination step that the order of the first operation using the first tool on the workpiece is correct and the next operation on the workpiece after the first operation is the second operation. When the computer generates the activation data for activating the operation of the second tool in the activation data generation step, a data transmission step of transmitting the activation data to the second tool; A control method including the above.

14. Learning data including image data obtained in advance by an imaging unit that performs imaging toward a work position, which is a position where a computer is performing work on a work as a processing target using a tool, the learning data including first image data obtained by imaging toward a first work position, which is the work position where a first operation is being performed on the work using a first tool as the tool, and second image data obtained by imaging toward a second work position, which is the work position where a second operation different from the first operation is being performed on the work using a second tool different from the first tool as the tool, a learned model for inferring the work position from new image data generated by machine learning using the learning data, the learned model being capable of classifying the first image data and the second image data by classification learning as the machine learning, a learned model acquisition step of acquiring the learned model; An inference data acquisition step in which the computer acquires the new image data obtained by imaging by the imaging unit as inference data; An inference result data output step in which, each time the computer acquires the inference data in the inference data acquisition step, the acquired inference data is input to the learned model and inference result data capable of specifying the work position is output; An operation order determination step in which the computer compares history data of the work position specified from the plurality of types of inference result data generated in the inference result data generation step with order data of the work position predetermined based on the order of operations on the work, to determine whether the order of operations on the work is correct; An activation data generation step in which, when the computer determines in the operation order determination step that the order of the first operation using the first tool on the work is correct and the operation following the first operation on the work is the second operation, the computer generates activation data for activating the operation of the second tool; When the computer generates the activation data for activating the operation of the second tool in the activation data generation step, a data transmission step of transmitting the activation data to the second tool; A control method including the above.

15. A computer, Learning data including image data obtained in advance by an imaging unit that performs imaging toward a work position, which is a position where work is being performed on a workpiece as a processing object using a tool, the learning data including first image data obtained by imaging toward a first work position, which is the work position where a first operation is being performed on the workpiece using a first tool as the tool, and second image data obtained by imaging toward a second work position, which is the work position where a second operation different from the first operation is being performed on the workpiece using a second tool different in type from the first tool, a learning data acquisition unit for acquiring the learning data; A learned model for inferring the work position from new image data by machine learning using the learning data, the learned model generation unit for generating the learned model capable of classifying the first image data and the second image data by classification learning as the machine learning; An inference data acquisition unit for acquiring the new image data obtained by imaging by the imaging unit as inference data; An inference result data generation unit for generating inference result data capable of specifying the work position by inputting the acquired inference data into the learned model each time the inference data acquisition unit acquires the inference data; A work order determination unit for determining whether the order of work on the workpiece is correct by comparing the history data of the work position specified from the plurality of types of inference result data generated by the inference result data generation unit with the order data of the work position predetermined based on the order of work on the workpiece; When the work order determination unit determines that the order of the first operation using the first tool on the workpiece is correct and the next operation after the first operation on the workpiece is the second operation, an activation data generation unit for generating activation data for activating the operation of the second tool; A data transmission unit for transmitting the activation data to the second tool when the activation data generation unit generates the activation data for activating the operation of the second tool; A program that functions as the above.

16. A learning model for inferring a working position from new image data generated by machine learning using learning data including image data preliminarily captured by an imaging unit that captures an image toward a working position where a computer is performing an operation on a workpiece as an object to be processed using a tool. The learning data includes first image data captured toward a first working position as the working position where a first operation is being performed on the workpiece using a first tool as the tool, and second image data captured toward a second working position as the working position where a second operation different from the first operation is being performed on the workpiece using a second tool as a tool of a type different from the first tool. The learning model is a learned model capable of classifying the first image data and the second image data by classification learning as the machine learning, and a learned model acquisition unit that acquires the learned model. An inference data acquisition unit that acquires the new image data captured by the imaging unit as inference data. An inference result data generation unit that, each time the inference data acquisition unit acquires the inference data, inputs the acquired inference data into the learned model to generate inference result data capable of specifying the working position. A work order determination unit that determines whether the work order for the workpiece is correct by comparing the history data of the working position specified from the plurality of types of inference result data generated by the inference result data generation unit with the order data of the working position predetermined based on the order of operations on the workpiece. An activation data generation unit that generates activation data for activating the operation of the second tool when the work order determination unit determines that the order of the first operation using the first tool on the workpiece is correct and the next operation after the first operation on the workpiece is the second operation. A data transmission unit that transmits the activation data to the second tool when the activation data generation unit generates the activation data for activating the operation of the second tool. A program that functions as. ​