Work system

Abstract

To provide a technique capable of outputting work information autonomously or in advance.SOLUTION: A work system includes: one or more fixed-point cameras; one or more carts each having an accommodation portion for accommodating at least one of a tool used for work and a component to be assembled to a work object, one or more cart-mounted cameras, and one or more information output devices for outputting work information; and a control device that controls operation of the information output device and has an acquisition portion for acquiring a captured image output from at least one of the fixed-point cameras and the cart-mounted cameras, a determination portion for determining which of a plurality of work processes the work represented by the captured image is on the basis of a position and a posture of an object represented by the captured image on an image by inputting the captured image to a trained machine learning model, and an output control portion for generating work information corresponding to the work process determined by the determination portion and transmitting the generated work information to the information output device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a work system. [Background technology]

[0002] Conventionally, there is known an information providing device that provides information to support training of a worker when the worker and a robot work together (Patent Document 1). In this technology, the information providing device provides a screen that accepts the selection of one process from a plurality of processes from a user. Then, the information providing device displays information corresponding to the process selected by the user on the screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-87705 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, information providing devices display information corresponding to a process selected by a user on a screen without using machine learning, making it difficult to autonomously generate and output information related to work or to output information in advance before work begins. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, there is provided a work system including: one or more carts each having one or more fixed cameras that capture bird's-eye images of a work site; one or more carts each having a storage unit that stores at least one of tools used in the work and parts to be assembled to a work target; one or more cart cameras that capture images of the cart and the area surrounding the cart; and an information output device that outputs work information related to the work to a worker using the cart at the work site, the control device having an acquisition unit that acquires the captured images output from at least one of the fixed cameras and the cart camera, a determination unit that inputs the captured images into a trained machine learning model and determines which of a plurality of work processes the work represented by the captured images is based on the position and orientation of an object represented by the captured images on an image; and an output control unit that generates the work information corresponding to the work process determined by the determination unit and transmits the generated work information to the information output device. According to this aspect, the work system can determine the work process using machine learning without the user having to select the work process. Therefore, the work system can autonomously generate work information corresponding to the work process currently being performed and output it from the information output device, or can proactively output work information corresponding to the next work process from the information output device before the next work process to be performed after the currently being performed work process is started. (2) In the above aspect, the determination unit determines which of the plurality of work processes the work represented by the captured image is, using an acquired image obtained by inputting the captured image into a trained machine learning model, the acquired image representing the position and the posture of the object represented by the captured image on the image, and when the object represented by the captured image is at least one of the cart, the tool, and the part, the acquired image is a segmentation image obtained by segmenting an area constituting the captured image into the object and an area other than the object, and the object represented by the captured image is the work. In a case where the captured image is a keypoint image representing the musculoskeletal structure of the worker detected by keypoint detection, the output control unit may generate the work information indicating the work procedure in the work process determined by the determination unit, the work information including at least one of the following information: the position of the cart, the type of tool used in the work, the position of the tool used in the work in the storage unit, the type of part to be assembled to the work object, the position of the part to be assembled to the work object in the storage unit, the assembly position of the part on the work object, and the worker's movement. According to this aspect, the work system can determine the work process using at least one of a segmentation image of the tool, a segmentation image of the part, and a keypoint image. The work system can output information indicating the work procedure in the work process determined as the work represented by the captured image from an information output device. This allows the work system to promptly confirm the work procedure if the worker forgets the work procedure. Furthermore, the work system can have the worker perform the work while sequentially confirming the work procedure. (3) In the above aspect, the output control unit may generate the work information including information indicating differences in the work between the worker represented by the captured image and another worker different from the worker represented by the captured image. According to this aspect, the work system can output information indicating the differences from the other worker from the information output device. This allows the work system to support the worker in improving their work by making them aware of the differences from the other worker. (4) In the above aspect, the output control unit may generate the task information according to the worker represented by the captured image. According to this aspect, the task system can output the task information according to the worker represented by the captured image from the information output device. This allows the task system to improve the worker's workability. (5) In the above aspect, when an abnormality occurs in the work, the output control unit may generate the work information including information about the abnormality. According to this aspect, when an abnormality occurs in the work, the work system can output information about the abnormality from the information output device. The present disclosure can be realized in various forms other than the above-described operation system, such as a method for manufacturing an operation system, a method for controlling an operation system, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the configuration of a work system. [Figure 2] FIG. 4 is a state transition diagram showing transitions in the operation state of the carriage. [Figure 3] FIG. 2 is a block diagram showing the configuration of a central control device. [Figure 4] FIG. 10 is a diagram showing an example of an operation flow of the work system. [Figure 5] FIG. 10 is a diagram showing an example of a work screen. [Figure 6] FIG. 10 is a diagram showing another example of the work screen. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: FIG. 1 is a diagram illustrating the configuration of a work system 1. The work system 1 is used by a worker P at a work location A to perform work on work objects W1 and W2. In this embodiment, the work objects W1 and W2 are vehicles. The work location A is a factory that manufactures vehicles. The factory includes, for example, a work line L and a preparation area R. On the work line L, the work objects W1 and W2 are transported by transportation equipment such as a conveyor. The worker P assembles predetermined parts using predetermined assembly tools at predetermined assembly positions on the work objects W1 and W2 transported on the work line L. In the preparation area R, at least one of the tools to be used for the work and the parts to be assembled on the work objects W1 and W2 is prepared on a cart 40. The worker P opens a shelf installed in the preparation area R that stores tools and parts, and grasps and removes the desired tool or part. Then, the worker P sets the tools and parts that he or she has taken out in predetermined positions on the cart 40 and closes the shelf. That is, in this embodiment, the work is performed by a work method in which the selection and assembly of parts are separated. The work method in which the selection and assembly of parts are separated is also called "SPS (Set Parts Supply)."

[0009] The work system 1 includes one or more fixed cameras 21 to 26, one or more carts 40, and a central control device 60.

[0010] The fixed cameras 21-26 capture images of the work place A from above. The fixed cameras 21-26 are equipped with communication devices (not shown) and can communicate with other devices such as the central control device 60 via wired or wireless communication. The fixed cameras 21-26 are fixed to support members such as the ceiling, walls, and pillars of the work place A, for example.

[0011] FIG. 2 is a state transition diagram showing transitions in the operating state of the dolly 40. The dolly 40 has a start mode, a standby mode, an operation mode, and a stop mode. When a start command is received from the worker P, the dolly 40 is set to the start mode. In the start mode, the dolly 40 starts up. When the dolly 40 is set to the start mode and becomes operable, the dolly 40 is set to the standby mode. In the standby mode, the dolly 40 waits to receive an operation from the worker P. When an operation to switch to the operation mode is received from the worker P during a period when the dolly 40 is set to the standby mode, the dolly 40 is set to the operation mode. In the operation mode, the dolly 40 can move by unmanned operation. "Unmanned operation" means operation without operation by the worker P. The operation by the worker P means an operation related to at least one of "running," "turning," and "stopping" of the dolly 40. Unmanned operation is achieved by automatic or manual remote control using a device located outside the carriage 40, or by autonomous control of the carriage 40. In this embodiment, during a period when unmanned operation control is being executed, the carriage 40 moves under remote control from the central control device 60. During a period when the carriage 40 is set to the operation mode, if an operation of the carriage 40 from the worker P is received, the carriage 40 is set to a standby mode. As a result, the carriage 40 performs an operation according to the operation of the worker P. In other words, manual operation by the worker P takes priority over automatic operation under unmanned operation control. During a period when the carriage 40 is set to the operation mode, if an abnormality occurs in the work, the carriage 40 temporarily stops and is set to a standby mode. During a period when the carriage 40 is set to the standby mode, if an end command is received from the worker P, the carriage 40 is set to an end mode. In the end mode, the power of the carriage 40 is cut off, and the carriage 40 comes to a complete stop.

[0012] As shown in FIG. 1, the cart 40 includes a storage section 41 , one or more cart cameras 43 , one or more information output devices 45 , and a cart control device 47 .

[0013] The storage section 41 stores at least one of tools used in the work and parts to be assembled to the work targets W1 and W2.

[0014] The dolly camera 43 captures images of the dolly 40 and the area surrounding the dolly 40. The dolly camera 43 can capture images of the environment surrounding the dolly 40, such as the state of tools and parts in the storage section 41, the movements of the worker P, and any interfering objects. The dolly camera 43 is equipped with a communication device (not shown) and can communicate with other devices such as the central control device 60 via wired or wireless communication. The dolly camera 43 is fixed to the outside of the dolly body 49, for example.

[0015] The information output device 45 outputs work information received from the central control device 60 to a worker P who is working using the cart 40 at work site A. The work information is information related to the work. The information output device 45 is, for example, a monitor or a speaker. The information output device 45 is, for example, fixed to the outside of the cart body 49.

[0016] The bogie control device 47 controls a group of actuators (not shown) to move the bogie 40. In this embodiment, the bogie control device 47 controls the group of actuators using a control signal received from the central control device 60 to move the bogie 40. The control signal includes, for example, at least one of the acceleration and the velocity of the bogie 40 and the steering angle as parameters.

[0017] FIG. 3 is a block diagram showing the configuration of the central control device 60. The central control device 60 controls the operation of the cart 40. In this embodiment, the central control device 60 is a server installed in a factory. The central control device 60 includes a processor 601, a memory 602, an input / output interface 603, and a bus 604. The processor 601, the memory 602, and the input / output interface 603 are connected via the bus 604 to enable bidirectional communication. A communication device 605 is connected to the input / output interface 603 for communicating with various devices external to the central control device 60. The communication device 605 can communicate with the cart control device 47 and the cart camera 43 via wireless communication and with the fixed cameras 21 to 26 via wired or wireless communication. The processor 601 executes a program PG stored in the memory 602 to function as an acquisition unit 611, a determination unit 612, an operation control unit 613, and an output control unit 614.

[0018] The acquisition unit 611 acquires the captured images output from at least one of the fixed cameras 21 to 26 and the dolly camera 43.

[0019] The determination unit 612 determines which of a plurality of work processes is performed on one work object W1, W2, the work represented by the captured image is, using an acquired image that shows the position and posture of the object represented by the captured image on the image. In this embodiment, when the object represented by the captured image is at least one of the cart 40, a tool, a part, and the work object W1, W2, the acquired image is a segmentation image in which the areas constituting the captured image are separated by segmentation into the target object and non-target object. When the object represented by the captured image is a worker P, the acquired image is a keypoint image that shows the musculoskeletal system of the worker P detected by keypoint detection.

[0020] In this embodiment, the determination unit 612 acquires an acquired image by inputting the captured image acquired by the acquisition unit 611 to a trained first machine learning model M1. The first machine learning model M1 is a machine learning model that outputs an acquired image when a captured image is input. When the object represented by the captured image is at least one of the cart 40, a tool, a part, and the work targets W1 and W2, the first machine learning model M1 is trained to perform, for example, instance segmentation. When the object represented by the captured image is a worker P, the first machine learning model M1 is trained to perform keypoint detection. The first machine learning model M1 can be, for example, a convolutional neural network (hereinafter, CNN) trained by supervised learning using a first training dataset D1. The first training dataset D1 includes, for example, a plurality of training images including at least one of the cart 40, a tool, a part, the worker P, and the work targets W1 and W2, and correct labels according to the types of objects represented by the training images. In the first training dataset D1, training images representing at least one of the cart 40, the tool, the part, and the work objects W1, W2 are associated as correct labels with region labels indicating whether each region in the training image represents the target object or a region other than the target object. A training image representing a worker P is associated as a correct label with a musculoskeletal label indicating the musculoskeletal structure of the worker P. Each of the multiple training images may further be associated with a classification label for classifying the object represented by the training image. During CNN training, it is preferable to update the CNN parameters by backpropagation (error backpropagation) so as to reduce the error between the output result by the first machine learning model M1 and the label.

[0021] Furthermore, the determination unit 612 inputs acquired images output from the first machine learning model M1 that represent at least one of a tool, a part, and a worker P into the trained second machine learning model M2. As a result, the determination unit 612 acquires process identification information indicating which of multiple work processes the work represented by the captured image from which the acquired image was derived is. The determination unit 612 determines that the work process identified by the process identification information is the work process represented by the captured image. The second machine learning model M2 is a machine learning model that outputs process identification information indicating the work process represented by the acquired image when the acquired image is input. The second machine learning model M2 can be, for example, a CNN trained by supervised learning using a second training dataset D2. The second training dataset D2 includes, for example, multiple acquired images as training images and process identification information as correct labels. During CNN training, it is preferable to update the parameters of the CNN using backpropagation (error backpropagation) to reduce errors between the output results of the second machine learning model M2 and the labels.

[0022] In this embodiment, the second training data set D2 includes training images that represent the position and posture of an object when an abnormality occurs in the work. "When an abnormality occurs in the work" refers to, for example, when the type of tool or part, the part installation position, or the part installation method is incorrect. Each of the multiple training images is further associated with status information that indicates the status of the work as supplementary information. The status information is information that indicates whether an abnormality occurs in the work.

[0023] Furthermore, in this embodiment, among the multiple training images, a training image representing a worker P is further associated with worker information about the worker P as supplementary information. The worker information includes, for example, at least one of worker identification information that identifies the multiple workers P, proficiency information that indicates the proficiency of the work, work content information that indicates the work content, and required time information that indicates the time required for the work. The proficiency information includes, for example, at least one of worker classification information and period information. The worker classification information is information that indicates whether the worker has been engaged in the work for a predetermined period or longer. The worker classification information is information that indicates, for example, whether the worker is a beginner or an expert. The period information is information that indicates the period of time that the worker has been engaged in the work. The period information is, for example, information that indicates the worker P's length of employment or years of experience. If the period specified by the period information is shorter than the predetermined period, the worker P is classified, for example, as a beginner. If the period specified by the period information is longer than the predetermined period, the worker P is classified, for example, as an expert. The worker information may include at least one of physique information indicating the physique of the worker P and characteristic information indicating the characteristics of the worker P, such as dominant hand.

[0024] The second training dataset D2 may include training images representing the positions and postures of the work objects W1 and W2 when work on each work object W1 and W2 begins. Specifically, the second training dataset D2 may include, as training images, segmentation images representing the work objects W1 and W2 when work on each work object W1 and W2 begins. The second training dataset D2 may include, as training images, segmentation images representing the cart 40 positioned at a preset initial position corresponding to the work on each work object W1 and W2. The second training dataset D2 may include, as training images, keypoint images representing the worker P waiting at a preset standby position corresponding to the work on each work object W1 and W2. Among the multiple training images, a training image representing the position and posture of an object when work begins is associated, for example, with process identification information indicating that the work is the first of multiple work processes performed on one work object W1 and W2 as a correct answer label. Among the plurality of training images, a training image showing the position and posture of an object at the start of a task may further be associated with object identification information as supplementary information. The object identification information is information indicating which of the plurality of work objects W1, W2 the task shown in the training image is being performed on.

[0025] Each of the training images may further be associated with at least one of the following additional information: type information indicating the type of the work objects W1, W2; object information regarding the work objects W1, W2; and requirement information indicating the requirements for the work. In the present embodiment, when the work objects W1, W2 are vehicles, the type information may be information indicating, for example, the vehicle class determined based on the overall length, width, and height, or the vehicle model classified based on the vehicle's external shape. The type information may also be information indicating factors that affect the work procedure, such as the drive system of the motor, engine, etc., or the type and installation location of accessories such as a steering wheel.

[0026] The operation control unit 613 controls the operation of the cart 40. In this embodiment, the operation control unit 613 controls the operation of the cart 40 by generating a control signal for controlling the operation of the cart 40 and transmitting it to the cart 40. During a period in which the cart 40 is set to the standby mode, the operation control unit 613 controls the operation of the cart 40 so as to perform an operation in accordance with an operation by the worker P. During a period in which the cart 40 is set to the operation mode, the operation control unit 613 moves the cart 40 to a position in accordance with the work process determined by the determination unit 612.

[0027] The output control unit 614 generates work information corresponding to the work process determined by the determination unit 612 and transmits the generated work information to the information output device 45. The output control unit 614 generates work information including, for example, procedure information indicating a work procedure in the work process determined by the determination unit 612. The procedure information includes, for example, at least one of the following information: the position of the cart 40, the type of tool used in the work, the position of the tool used in the work in the storage unit 41, the type of parts to be assembled to the work objects W1, W2, the position of the parts to be assembled to the work objects W1, W2 in the storage unit 41, the assembly position of the parts on the work objects W1, W2, and the movement of the worker P. The output control unit 614 may generate work information including difference information indicating differences in the work between the worker P represented in the captured image and another worker P different from the worker P represented in the captured image. The difference information is, for example, information indicating differences from an experienced worker. The output control unit 614 may generate work information according to the worker P represented by the captured image. When an abnormality occurs in the work, the output control unit 614 may generate work information including abnormality information related to the abnormality.

[0028] Fig. 4 is a diagram showing an example of an operation flow of the work system 1. The flow shown in Fig. 4 starts, for example, when the power of the cart 40 is turned on.

[0029] In step S1, worker P signals to cameras 21-26, 43 to start the cart 40 in order to start work on the first work object W1, which is being transported as the Nth object on the work line L. The central control device 60 acquires captured images from fixed cameras 21-26 and the cart camera 43, which include worker P in their detection ranges. The central control device 60 acquires keypoint images by inputting the captured images representing worker P into a trained first machine learning model M1. When information representing a predetermined posture is acquired from the keypoint images, the central control device 60 detects that worker P has signaled to cameras 21-26, 43 to start the cart 40. This causes the central control device 60 to accept a start command from worker P for the cart 40. When the central control device 60 accepts a start command from worker P for the cart 40, it sets the cart 40 to a start mode. This causes the cart 40 to start. When the carriage 40 becomes operable, the central control unit 60 sets the carriage 40 to a standby mode.

[0030] In step S2, the worker P moves the cart 40 to a standby position corresponding to the work performed on the first work object W1 in order to move the cart 40 to an initial position corresponding to the work performed on the first work object W1. The central control device 60 detects that the worker P has moved to the standby position corresponding to the work performed on the first work object W1. When the central control device 60 detects that the worker P has moved to the standby position corresponding to the work performed on the first work object W1, the central control device 60 extracts, from the second training dataset D2, a segmentation image of the cart 40 associated with process identification information indicating that this is the work process to be performed first and object identification information indicating the first work object W1. Based on the position and orientation of the cart 40 on the extracted segmentation image, the central control device 60 generates a control signal for moving the cart 40 to an initial position corresponding to the work performed on the first work object W1 and transmits the generated control signal to the cart 40. The carriage control device 47 controls the actuator group using the control signal received from the central control device 60, thereby moving the carriage 40 to an initial position appropriate for the work on the first work object W1. The central control device 60 sets the carriage 40 to an operating mode. This causes the carriage control device 47 to start controlling unmanned operation. The worker P uses the carriage 40 to start work on the first work object W1.

[0031] In step S3, the central control device 60 acquires captured images from the dolly camera 43, whose detection range includes at least one of a tool used in the work on the first work object W1 and a part to be assembled to the first work object W1. The central control device 60 acquires captured images from the fixed cameras 21-26 and the dolly camera 43, whose detection range includes the worker P. The central control device 60 inputs the captured images representing at least one of the tool, part, and worker P into the trained first machine learning model M1 to acquire captured images representing at least one of the tool, part, and worker P. The central control device 60 inputs the captured images representing at least one of the tool, part, and worker P into the trained second machine learning model M2 to acquire process identification information. The central control device 60 determines that the work process identified by the process identification information is the work process represented by the captured images. The central control device 60 extracts a segmentation image of the dolly 40 associated with the acquired process identification information from the second training dataset D2. The central control device 60 determines the next target position to which the cart 40 should head, based on the position and orientation of the cart 40 on the extracted segmentation image. The central control device 60 generates a control signal for moving the cart 40 to the determined target position, and transmits the generated control signal to the cart 40. The cart control device 47 controls the actuator group using the control signal received from the central control device 60, thereby moving the cart 40 to a position appropriate for the work process.

[0032] During the period when the carriage 40 is set to the operation mode, the central control device 60 repeatedly identifies the work process, determines the target position according to the work process, generates control signals, and transmits the control signals at a predetermined cycle. The carriage control device 47 repeatedly receives control signals and controls the actuators at a predetermined cycle. As a result, the work system 1 successively corrects the position of the carriage 40 even during work.

[0033] In step S4, the central control device 60 generates work information corresponding to the work process determined to be the work represented by the captured image, and transmits the generated work information to the information output device 45. The information output device 45 outputs the received work information as images, text, audio, etc. This provides the worker P with procedure information and difference information.

[0034] In step S5, while the dolly 40 is set to the operation mode, the worker P signals to the cameras 21-26, 43 to manually operate the dolly 40 as necessary. As in step S1, the central control device 60 detects the signal from the worker P through image processing and accepts an operation from the worker P regarding the dolly 40. When the central control device 60 accepts an operation from the worker P regarding the dolly 40, the central control device 60 identifies the operation requested by the worker P regarding the dolly 40 from the operation content of the worker P. The central control device 60 generates a control signal for executing the identified operation and transmits the generated control signal to the dolly 40. The dolly control device 47 controls the actuator group using the control signal received from the central control device 60 to execute the operation requested by the worker P. When the central control device 60 accepts an operation from the worker P regarding the dolly 40 while the dolly 40 is set to the operation mode, the central control device 60 sets the dolly 40 to the standby mode.

[0035] Furthermore, if an abnormality occurs in the work while the cart 40 is set to the operation mode, the central control device 60 generates a control signal to temporarily halt the cart 40 and transmits the generated control signal to the cart 40. The cart control device 47 temporarily halts the cart 40 by controlling the actuator group using the control signal received from the central control device 60. If an abnormality occurs in the work while the cart 40 is set to the operation mode, the central control device 60 sets the cart 40 to standby mode. If an abnormality occurs in the work, the central control device 60 generates work information including abnormality information and transmits the generated work information to the information output device 45. The information output device 45 outputs the received work information as an image, text, audio, or the like. This provides the abnormality information to the worker P. Note that if an abnormality occurs in the work, the cart 40 may be temporarily halted by manual operation by the worker P.

[0036] The central control device 60 periodically identifies the work process, detects anomalies, generates work information according to the work process and the detected anomaly, and transmits the work information. The information output device 45 periodically receives work information and outputs the work information. This allows the work system 1 to update the work information it outputs each time.

[0037] In step S6, when the worker P wants to resume the unmanned movement of the cart 40, the worker P gives a predetermined signal toward the cameras 21-26, 43. As in step S1, the central control device 60 detects the signal from the worker P by image processing and accepts an operation to resume the unmanned movement. When the central control device 60 accepts an operation to resume the unmanned movement of the cart 40, the central control device 60 sets the cart 40 to the operating mode again. This causes the cart control device 47 to resume control of the unmanned operation. Note that when the cart 40 resumes its unmanned movement, the worker P may give a signal toward the cameras 21-26, 43 to cause the cart 40 to perform a desired operation.

[0038] In step S7, when all work processes for the first work object W1 are completed, the worker P moves to a standby position corresponding to the work for the second work object W2 to begin work on the second work object W2, which is being transported (N+1)th on the work line L. The central control device 60 detects that the worker P has moved to the standby position corresponding to the work for the second work object W2. When the central control device 60 detects that the worker P has moved to the standby position corresponding to the work for the second work object W2, it sets the cart 40 to an operating mode. Then, the central control device 60 extracts, from the second training data set D2, a segmentation image of the cart 40 associated with process identification information indicating that this is the work process to be performed first and object identification information indicating the second work object W2. Based on the position and orientation of the cart 40 on the extracted segmentation image, the central control device 60 generates a control signal for moving the cart 40 to an initial position corresponding to the work for the second work object W2 and transmits the generated control signal to the cart 40. The carriage control device 47 controls the actuator group using the control signal received from the central control device 60, thereby moving the carriage 40 to an initial position according to the work on the second work object W2. The worker P starts work on the second work object W2 using the carriage 40.

[0039] FIG. 5 is a diagram illustrating an example of a work screen displayed on a monitor serving as the information output device 45. FIG. 5 illustrates a work screen in which work is progressing normally without any abnormalities during a period when the cart 40 is set to the operating mode. The central control device 60 extracts, from the second training data set D2, acquired images associated with process identification information indicating the work process determined to be the work represented by the captured images. The central control device 60 generates procedure information I1 based on the position and orientation of the object in the extracted acquired images. Furthermore, the central control device 60 identifies the worker P represented by the captured images using pattern recognition or the like. The central control device 60 extracts, from the second training data set D2, proficiency information associated with the worker identification information indicating the identified worker P. The central control device 60 uses the extracted proficiency information to classify the worker P represented by the captured images as either a beginner or an expert. When the central control device 60 classifies the worker P represented by the captured image as a beginner, it extracts, from the second training dataset D2, task content information and required time information associated with the worker identification information indicating the identified worker P and the worker identification information indicating the worker P classified as an expert. The central control device 60 uses the extracted task content information and required time information to generate difference information I2 indicating the difference in task time between the identified worker P and an expert for the same task content. The central control device 60 may also generate task information I3 including other information such as task time information, type information, vehicle information as object information, and requirement information. The task time information is, for example, information indicating at least one of a predetermined task target time, a remaining time relative to the task target time, and a delay time relative to the task target time.

[0040] FIG. 6 is a diagram showing another example of a work screen displayed on a monitor serving as the information output device 45. FIG. 6 shows a work screen when an abnormality occurs in work during a period when the cart 40 is set to the operation mode. The central control device 60 detects that an abnormality has occurred in work, for example, by comparing an acquired image when the work is progressing normally without any abnormalities with an acquired image output from the first machine learning model M1. When the central control device 60 detects that an abnormality has occurred in work, it generates, as anomaly information, abnormality occurrence information I4 indicating that an abnormality has occurred in work. Furthermore, the central control device 60 generates, as anomaly information, indication information I5 for resolving the abnormality or improving the work of the worker P.

[0041] According to the above embodiment, the work system 1 can use machine learning to determine which of multiple work processes the work represented by a captured image corresponds to, based on the position and orientation of the object represented in the captured image. The work system 1 can then generate work information corresponding to the work process determined to represent the work represented by the captured image and output it from the information output device 45 of the cart 40. In other words, the work system 1 can use machine learning to determine the work process without the user having to select the work process. Therefore, the work system 1 can autonomously generate work information corresponding to the work process currently being performed and output it from the information output device 45, or can proactively output work information corresponding to the next work process from the information output device 45 before the next work process to be performed after the currently being performed work process is started.

[0042] Furthermore, according to the above embodiment, the work system 1 can determine which of a plurality of work processes the work represented by the captured image corresponds to, by using at least one of a segmentation image of a tool or part and a keypoint image representing the musculoskeletal structure of the worker P. Then, the work system 1 can output, from the information output device 45, procedure information I1 indicating the work procedure in the work process determined as the work represented by the captured image. As a result, if the worker P forgets the work procedure, the work system 1 can promptly have the worker P confirm the work procedure.

[0043] Furthermore, according to the above embodiment, the work system 1 can update the work information to be output each time. In this way, the work system 1 can have the worker P work while checking the work procedure as it goes along. This reduces the possibility of an abnormality occurring in the work.

[0044] Furthermore, according to the above embodiment, the work system 1 can output difference information I2 indicating differences between the worker P and other workers P from the information output device 45. This allows the work system 1 to support work improvement by making the worker P aware of the differences between the worker P and other workers P. In particular, when the worker P depicted in the captured image is a beginner, the work system 1 can support work improvement by outputting difference information I2 indicating differences between the worker P and an expert from the information output device 45, thereby enabling the beginner to master the work more quickly.

[0045] Furthermore, according to the above embodiment, the work system 1 can output work information according to the physique and characteristics of the worker P depicted in the captured image from the information output device 45. This allows the work system 1 to improve the workability of the worker P. Note that if the worker P depicted in the captured image is a beginner, the work system 1 may output standard movements of an expert or the like from the information output device 45. In this way, the work system 1 can teach the beginner the standard movements.

[0046] Furthermore, according to the above embodiment, the work system 1 can detect that an abnormality has occurred in the work, and output work information including the abnormality information from the information output device 45. This allows the work system 1 to notify the worker P that an abnormality has occurred in the work, and to suggest to the worker P how to resolve the abnormality. Furthermore, when the work system 1 detects that an abnormality has occurred in the work, it can temporarily stop the cart 40.

[0047] Furthermore, according to the above embodiment, type information is associated with each of the multiple training images as supplementary information, so that the work system 1 can output work information according to the type of work objects W1, W2 from the information output device 45.

[0048] B. Other Embodiments: (B1) The work objects W1 and W2 may be other than vehicles. The work objects W1 and W2 may be moving bodies other than vehicles, or objects other than moving bodies. Furthermore, when the work objects W1 and W2 are moving bodies, the moving bodies may be capable of moving by unmanned operation. In this case, the moving bodies are transported on the work line L by utilizing the movement of the moving bodies by unmanned operation.

[0049] (B2) Work location A may be a location other than a factory that manufactures work objects W1 and W2. Work location A may be a factory that repairs work objects W1 and W2 after shipment, a factory that inspects work objects W1 and W2 after shipment, or a location other than a factory.

[0050] (B3) The carriage 40 may be moved by autonomous control. In this case, the functions of the central control device 60 are realized by the carriage control device 47. The carriage 40 may also be moved only by manual operation, without being moved by unmanned operation.

[0051] (B4) The machine learning models M1 and M2 may be updated by re-learning after the work is completed. This improves the accuracy of the machine learning models M1 and M2. Therefore, the central control device 60 can output more appropriate work information from the information output device 45 at the necessary timing depending on the work situation, or move the cart 40 to a more appropriate position at the necessary timing. Note that even if an abnormality in the work is detected, if it is a false detection, the machine learning models M1 and M2 may be re-learned using the image in which the abnormality in the work was falsely detected as a training image. Whether or not an abnormality in the work was correctly detected can be determined, for example, by whether or not the operator P operates the operation button B1 displayed on the monitor serving as the information output device 45. In this configuration, the work system 1 can accurately detect an abnormality in the work.

[0052] (B5) The central control device 60 may accept operations from the worker P on the trolley 40 by recognizing that a button provided on the trolley body 49 or the work location A, or a stop button B2 displayed on a monitor serving as the information output device 45, has been operated.

[0053] (B6) At least some of the work performed in the preparation area R may be performed by a robot, or may be performed through collaboration between a worker P and a robot. In addition, the work may be performed using a work method other than SPS.

[0054] (B7) The determination unit 612 may input the captured image into a trained third machine learning model to acquire process identification information indicating which of multiple work processes the work represented by the captured image is, and determine that the work process identified by the process identification information is the work process represented by the captured image. The third machine learning model is a machine learning model that outputs process identification information indicating the work process represented by the captured image when the captured image is input. For example, the third machine learning model may be a CNN trained by supervised learning using a third training dataset. The third training dataset includes various information contained in the first training dataset D1 and the second training dataset D2. The third training dataset may include, for example, a plurality of training images including at least one of the cart 40, tools, parts, worker P, and work targets W1 and W2, and process identification information as a correct label. In this configuration, the determination unit 612 can determine the work process without acquiring the captured image.

[0055] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0056] 1...work system, 21-26...fixed camera, 40...cart, 41...storage section, 43...cart camera, 45...information output device, 47...cart control device, 49...cart body, 60...central control device, 601...processor, 602...memory, 603...input / output interface, 604...bus, 605...communication device, 611...acquisition section, 612...determination section, 613...operation control section, 614...output control section, A...work location, B1...operation button, B2...stop button, D1...first learning dataset, D2...second learning dataset, I1...procedure information, I2...difference information, I3...other information, I4...abnormality occurrence information, I5...indication information, L...work line, M1...first machine learning model, M2...second machine learning model, P...worker, PG...program, R...preparation area, W1...first work object, W2...second work object

Claims

1. 1. A work system comprising: One or more fixed cameras that capture an overhead image of the work place; One or more carriages, a storage section for storing at least one of a tool used in the work and a part to be assembled to the work object; one or more dolly cameras that capture images of the dolly and the area surrounding the dolly; one or more carts each having one or more information output devices that output work information related to the work to a worker using the cart at the work site; A control device for controlling an operation of the information output device, an acquisition unit that acquires captured images output from at least one of the fixed camera and the dolly camera; a determination unit that determines which of a plurality of work processes the work represented by the captured image is based on the position and orientation of the object represented by the captured image on the image by inputting the captured image into a trained machine learning model; A work system comprising: a control device having an output control unit that generates work information according to the work process determined by the determination unit and transmits the generated work information to the information output device.

2. The work system according to claim 1, the determination unit determines which of the plurality of work processes the work represented by the captured image is, using an acquired image that is acquired by inputting the captured image into a trained machine learning model and that represents the position and the orientation of the object represented by the captured image on the image; When the object represented in the captured image is at least one of the carriage, the tool, and the part, the acquired image is a segmentation image obtained by separating an area constituting the captured image into the object and an area other than the object by segmentation, When the object represented by the captured image is the worker, the acquired image is a keypoint image representing a musculoskeletal structure of the worker detected by keypoint detection, The output control unit generates the work information, which is information indicating the work procedure in the work process determined by the determination unit, and includes at least any of the following information: the position of the cart, the type of tool used in the work, the position of the tool used in the work within the storage unit, the type of part to be assembled to the work object, the position of the part to be assembled to the work object within the storage unit, the assembly position of the part on the work object, and the movement of the worker.

3. The work system according to claim 1, The output control unit generates the work information including information indicating differences in the work between the worker represented by the captured image and another worker different from the worker represented by the captured image.

4. The work system according to claim 1, The output control unit generates the work information according to the worker represented by the captured image.

5. The work system according to claim 1, When an abnormality occurs in the work, the output control unit generates the work information including information related to the abnormality.

Images