Working system

Abstract

Provided is a work system capable of automatically moving a trolley for work in accordance with work conditions. This work system is provided with: one or more fixed-point cameras for capturing an overhead image of a work site; the one or more trolleys can move through unmanned driving, and each trolley is provided with a containing part for containing a tool used for work and a component assembled to a work object; and one or more trolley cameras that capture images of the trolley and a peripheral area of the trolley. And a control device for controlling the operation of the carriage, the control device comprising: an acquisition unit for acquiring a captured image; a determination unit that, by inputting the captured image into the learned machine learning model, determines which of the plurality of work steps the work indicated by the captured image is on the basis of the position and orientation of the object on the image indicated by the captured image; and an operation control unit that moves the trolley to a position corresponding to the work process determined by the determination unit.

Description

Technical Field

[0001] This disclosure relates to operating systems. Background Technology

[0002] Previously, there were known follow vehicles that followed moving guides such as people, vehicles, movable robots, and autonomous vehicles (Patent Document 1). In this technology, the follow vehicle uses sensor data output from sensors such as cameras to calculate the position of the guide and travels by following the guide.

[0003] Patent Document 1: Japanese Patent Publication No. 2023-537700. Summary of the Invention The technical problem that the invention aims to solve

[0004] In conventional technologies, following vehicles do not analyze the movement and configuration of objects outside the guide body in the driving area, nor do they analyze the surrounding conditions of the following vehicle in a time-series manner to appropriately calculate the position of the guide body before driving. Therefore, following vehicles struggle to drive autonomously based on their surrounding conditions or to anticipate the upcoming actions of movable objects such as workers in the driving area. This technical problem is not limited to following vehicles; it is also common in trolleys used to house and transport tools and components at work sites where work is being performed. Means for solving technical problems

[0005] This disclosure can be implemented in the following ways.

[0006] According to one aspect of this disclosure, a work system is provided. The work system includes: one or more fixed-point cameras for overlooking and photographing the work site; one or more trolleys capable of moving autonomously and having: a housing for housing tools used in the work and components for assembly onto the work object; and one or more trolley cameras for photographing the trolleys and their surrounding area; and a control device for controlling the movement of the trolleys and having: an acquisition unit for acquiring images output from at least one of the fixed-point cameras and the trolley cameras; a judgment unit for determining, by inputting the photographed images into a learned machine learning model, which of a plurality of work processes the work represented by the photographed images is based on the position and posture of the object represented by the photographed images in the image; and a motion control unit for moving the trolleys to a position corresponding to the work process determined by the judgment unit. According to this method, the control device can analyze the work status in the work site in a time-series manner based on the movements and configurations of multiple objects present in the work site. Therefore, the control device can automatically move the trolley to the appropriate position when needed, based on the working conditions, or predict the next actions of movable objects such as workers in the work area and move the trolley in advance. This disclosure can be implemented in various ways other than the operating system described above. For example, it can be implemented as a method for manufacturing an operating system, a method for controlling an operating system, a computer program for implementing the control method, or a non-transitory recording medium on which the computer program is recorded. Attached Figure Description

[0007] Figure 1 This is a diagram showing the structure of the operating system. Figure 2 This is a block diagram showing the structure of the central control unit. Figure 3 This is a diagram illustrating an example of the operational flow of a work system. Detailed Implementation

[0008] A. First implementation method: Figure 1This diagram illustrates the structure of the work system 1. The work system 1 is used by operator P to perform work on work objects W1 and W2 at work site A. In this embodiment, the work objects W1 and W2 are vehicles. Work site A is a vehicle manufacturing plant. The plant includes, for example, a production line L and a preparation area R. On the production line L, work objects W1 and W2 are transported by conveyor or other transport equipment. Operator P uses pre-defined tools to assemble pre-defined components into pre-defined assembly positions on the work objects W1 and W2 transported on the production line L. In the preparation area R, at least one of the tools for the work and the components to be assembled on the work objects W1 and W2 is prepared on a trolley 40. Operator P opens the shelf in the preparation area R where tools and components are stored, grasps and removes the desired tools and components. Operator P then places the removed tools and components in a pre-defined position on the trolley 40 and closes the shelf. That is, in this embodiment, the work is performed by separating the selection and assembly of components. The work method that separates the selection and assembly of parts is also called "SPS (Set PartsSupply)".

[0009] The operating system 1 has one or more fixed cameras 21-26, one or more trolleys 40 and a central control device 60.

[0010] Fixed-point cameras 21-26 take aerial views of work site A. Fixed-point cameras 21-26 are equipped with communication devices (not shown) and can communicate with other devices such as the central control unit 60 via wired or wireless communication. Fixed-point cameras 21-26 are, for example, fixed to supporting components such as the ceiling, walls, or pillars of work site A.

[0011] The trolley 40 has a start mode, a standby mode, an operation mode, and an end mode. Upon receiving a start command from operator P, the trolley 40 is set to start mode. In start mode, the trolley 40 starts. While the trolley 40 is in start mode, if it becomes operational, it is set to standby mode. In standby mode, the trolley 40 remains idle to receive operations from operator P. While the trolley 40 is in standby mode, if it receives an operation from operator P to switch to operation mode, it is set to operation mode. In operation mode, the trolley 40 can move autonomously. "Autonomous operation" means driving without relying on the operation of operator P. The operation of operator P refers to operations related to at least one of "driving," "turning," or "stopping" of the trolley 40. Autonomous operation is achieved through automatic or manual remote control using a device located external to the trolley 40, or through the autonomous control of the trolley 40. In this embodiment, during unmanned operation, the trolley 40 moves via remote control from the central control unit 60. While the trolley 40 is in operating mode, upon receiving an operation from operator P, the trolley 40 is set to standby mode. Thereby, the trolley 40 performs actions corresponding to the operator P's operation. That is, operator P's manual operation takes priority over automatic operation based on unmanned control. While the trolley 40 is in operating mode, if an abnormality occurs during operation, the trolley 40 temporarily stops and is set to standby mode. While the trolley 40 is in standby mode, upon receiving a termination command from operator P, the trolley 40 is set to termination mode. In termination mode, the power to the trolley 40 is cut off, and the trolley 40 comes to a complete stop.

[0012] The trolley 40 has a housing 41, one or more trolley cameras 43, and a trolley control device 47.

[0013] The housing 41 houses at least one of the tools used for the operation and the components assembled to the objects W1 and W2.

[0014] The trolley camera 43 captures images of the trolley 40 and its surrounding area. For example, the trolley camera 43 can capture images of the status of tools and components within the housing 41, the actions of the operator P, and any interfering objects in the surrounding environment of the trolley 40. The trolley camera 43 has a communication device (not shown) and can communicate with other devices such as the central control unit 60 via wired or wireless communication. The trolley camera 43 is, for example, fixed to the exterior of the trolley body 49.

[0015] The trolley control device 47 moves the trolley 40 by controlling an actuator assembly (not shown). In this embodiment, the trolley control device 47 controls the actuator assembly using control signals received from the central control device 60, thereby moving the trolley 40. The control signals include, for example, at least one of the acceleration and speed of the trolley 40 and the steering angle as parameters.

[0016] Figure 2 This is a block diagram showing the structure of the central control unit 60. The central control unit 60 controls the operation of the trolley 40. In this embodiment, the central control unit 60 is a server installed in the factory. The central control unit 60 includes a processor 601, a memory 602, an input / output interface 603, and a bus 604. The processor 601, memory 602, and input / output interface 603 are connected via the bus 604 in a bidirectional communication manner. A communication device 605 for communicating with various external devices of the central control unit 60 is connected to the input / output interface 603. The communication device 605 can communicate wirelessly with the trolley control device 47 and the trolley camera 43, and can communicate with the fixed-point cameras 21-26 via wired or wireless communication. The processor 601 functions as an acquisition unit 611, a judgment unit 612, and an action control unit 613 by executing the program PG stored in the memory 602.

[0017] The acquisition unit 611 acquires images output from at least one of the fixed-point cameras 21-26 and the trolley camera 43.

[0018] The determination unit 612 uses an acquired image representing the position and posture of the object represented by the captured image to determine which of a plurality of work steps is performed on a work object W1 or W2, and which of the captured image represents the work process. In this embodiment, when the object represented by the captured image is at least one of the trolley 40, tool, component, and work object W1 or W2, the acquired image is obtained by segmenting the region constituting the captured image into the object object and a segmented image other than the object object. When the object represented by the captured image is the worker P, the acquired image is a keypoint image representing the musculoskeletal structure of the worker P detected by keypoint detection.

[0019] In this embodiment, the determination unit 612 acquires an acquired image by inputting the captured image acquired by the acquisition unit 611 into the first machine learning model M1 after it has been trained. The first machine learning model M1 is a machine learning model that outputs the acquired image when the captured image is input. When the object represented by the captured image is at least one of the following: trolley 40, tool, component, and work object W1, W2, the first machine learning model M1 learns, for example, in a manner that performs instance segmentation. When the object represented by the captured image is the worker P, the first machine learning model M1 learns, for example, in a manner that performs keypoint detection. As the first machine learning model M1, a convolutional neural network (hereinafter referred to as CNN) that has been trained using a teacher-trained dataset D1 can be used, for example. The first learning dataset D1, for example, has multiple training images including at least one of the following: trolley 40, tool, component, worker P, and work object W1, W2, and correct answer labels corresponding to the types of objects represented by the training images. In the first training dataset D1, training images representing at least one of the following: trolley 40, tools, parts, and work objects W1 and W2, are associated with region labels as correct answer labels. Region labels indicate which region in the training image represents the object and which region is outside the object. Training images representing operator P are associated with musculoskeletal labels as correct answer labels. Musculoskeletal labels represent the muscles and bones of operator P. Furthermore, each training image can be associated with a classification label for classifying the object represented by the training image. During CNN learning, it is preferable to update the CNN parameters through backpropagation (error backpropagation method) to reduce the error between the output of the first machine learning model M1 and the label.

[0020] Furthermore, the determination unit 612 inputs the acquired image representing at least one of the tools, components, and operators P from the acquired image output by the first machine learning model M1 into the learned second machine learning model M2. Thus, the determination unit 612 obtains process identification information indicating which of a plurality of work processes the captured image, which serves as the basis for the acquired image, represents. The determination unit 612 determines the work process determined by the process identification information as the work process represented by the captured image. The second machine learning model M2 is a machine learning model that outputs process identification information representing the work process represented by the acquired image when the acquired image is input. For example, the second machine learning model M2 can be a CNN learned using a teacher-trained dataset D2. The second learning dataset D2, for example, has multiple acquired images as training images and process identification information as labels for correct answers. During CNN learning, it is preferable to update the CNN parameters through backpropagation (error backpropagation method) to reduce the error between the output of the second machine learning model M2 and the labels.

[0021] Additionally, the second learning dataset D2 may also include training images representing the position and posture of objects W1 and W2 at the start of the work for each work object W1 and W2. That is, the second learning dataset D2 may also include segmented images of work objects W1 and W2 at the start of the work for each work object W1 and W2 as training images. The second learning dataset D2 may also include segmented images of the trolley 40 positioned at a pre-set initial position according to the work for each work object W1 and W2 as training images. The second learning dataset D2 may also include keypoint images of the operator P waiting at a pre-set standby position according to the work for each work object W1 and W2 as training images. The training images representing the position and posture of objects at the start of the work are associated with, for example, process identification information as correct answer labels. This process identification information indicates the first of multiple work processes performed on a work object W1 and W2. The training images representing the position and posture of objects at the start of the task can be further associated with object recognition information as supplementary information. Object recognition information indicates which of the multiple task objects W1 and W2 the task represented by the training image is being performed on.

[0022] The motion control unit 613 controls the movement of the trolley 40. In this embodiment, the motion control unit 613 controls the movement of the trolley 40 by generating a control signal for its movement and sending it to the trolley 40. When the trolley 40 is in standby mode, the motion control unit 613 controls the movement of the trolley 40 to perform actions corresponding to the operation of the operator P. When the trolley 40 is in operation mode, the motion control unit 613 moves the trolley 40 to a position corresponding to the work procedure determined by the judgment unit 612.

[0023] Figure 3 This is a diagram illustrating an example of the operation flow of operating system 1. Figure 3 The process shown begins, for example, when the power to the trolley 40 is turned on.

[0024] In step S1, operator P, in order to begin work on the first work object W1 transported to the Nth workstation on work line L, sends a signal to the fixed-point cameras 21-26 and the trolley camera 43 to start the trolley 40. The central control unit 60 acquires images from the fixed-point cameras 21-26 and the trolley camera 43, which include operator P within their detection range. The central control unit 60 inputs the images representing operator P into the first machine learning model M1, which has completed its learning, thereby obtaining keypoint images. Having obtained information representing a pre-defined posture from the keypoint images, the central control unit 60 detects the signal sent by operator P to the fixed-point cameras 21-26 and the trolley camera 43 to start the trolley 40. Therefore, the central control unit 60 receives a start command from operator P for the trolley 40. Upon receiving the start command from operator P, the central control unit 60 sets the trolley 40 to start mode. Thus, the trolley 40 starts. When the trolley 40 becomes operational, the central control device 60 sets the trolley 40 to standby mode.

[0025] In step S2, operator P moves the trolley 40 to a standby position corresponding to the operation of the first work object W1 in order to move the trolley 40 to the initial position corresponding to the operation of the first work object W1. The central control unit 60 detects that operator P has moved to the standby position corresponding to the operation of the first work object W1. Upon detecting that operator P has moved to the standby position corresponding to the operation of the first work object W1, the central control unit 60 extracts a segmented image of the trolley 40 from the second learning dataset D2, which is associated with process identification information indicating the first executed operation and object identification information indicating the first work object W1. Based on the position and posture of the trolley 40 in the extracted segmented image, the central control unit 60 generates a control signal to move the trolley 40 to the initial position corresponding to the operation of the first work object W1, and sends the generated control signal to the trolley 40. The trolley control unit 47 uses the control signal received from the central control unit 60 to control the actuator group, thereby moving the trolley 40 to the initial position corresponding to the operation of the first work object W1. The central control unit 60 sets the trolley 40 to operating mode. Consequently, the trolley control unit 47 initiates unmanned operation. Operator P uses the trolley 40 to begin work on the first work object W1.

[0026] In step S3, the central control device 60 acquires images from a trolley camera 43 that includes at least one of the tools used for the operation on the first work object W1 and the components assembled onto the first work object W1 within its detection range. The central control device 60 also acquires images from fixed-point cameras 21-26 and the trolley camera 43, both of which include the operator P within its detection range. The central control device 60 obtains an acquired image representing at least one of the tools, components, and operator P by inputting the acquired image representing at least one of the tools, components, and operator P into a learned first machine learning model M1. The central control device 60 obtains process identification information by inputting the acquired image representing at least one of the tools, components, and operator P into a learned second machine learning model M2. The central control device 60 determines the work process identified by the process identification information as the work process represented by the captured image. The central control device 60 extracts a segmented image of the trolley 40 that is associated with the acquired process identification information from a second learning dataset D2. Based on the position and orientation of the trolley 40 in the extracted segmented image, the central control unit 60 determines the target location where the trolley 40 should go next. The central control unit 60 generates a control signal to move the trolley 40 to the determined target location and sends the generated control signal to the trolley 40. The trolley control unit 47 uses the control signal received from the central control unit 60 to control the actuator assembly, thereby moving the trolley 40 to the position corresponding to the work process.

[0027] In step S4, while the trolley 40 is set to the operating mode, the central control device 60 repeatedly determines the work sequence, decides the target position corresponding to the work sequence, generates control signals, and sends control signals at a predetermined cycle. The trolley control device 47 repeatedly receives control signals and controls the actuator assembly at a predetermined cycle. As a result, the work system 1 also corrects the position of the trolley 40 sequentially during operation.

[0028] In step S5, during the period when the trolley 40 is set to the operating mode, the operator P sends signals to the fixed-point cameras 21-26 and the trolley camera 43 as needed to manually operate the trolley 40. Similar to step S1, the central control unit 60 detects the signals from the operator P through image processing, thereby receiving the operation from the operator P on the trolley 40. Upon receiving the operation from the operator P on the trolley 40, the central control unit 60 determines the action requested by the operator P on the trolley 40 based on the content of the operation. The central control unit 60 generates a control signal to execute the determined action and sends the generated control signal to the trolley 40. The trolley control unit 47 uses the control signal received from the central control unit 60 to control the actuator assembly, thereby executing the action requested by the operator P. During the period when the trolley 40 is set to the operating mode, upon receiving the operation from the operator P on the trolley 40, the central control unit 60 sets the trolley 40 to the standby mode. Furthermore, during the period when the trolley 40 is set to the operating mode, in the event of an operational malfunction, the central control unit 60 generates a control signal to temporarily stop the trolley 40 and sends the generated control signal to the trolley 40. The trolley control unit 47 uses the control signal received from the central control unit 60 to control the actuator assembly, thereby temporarily stopping the trolley 40. During the period when the trolley 40 is set to the operating mode, in the event of an operational malfunction, the central control unit 60 sets the trolley 40 to standby mode. Additionally, in the event of an operational malfunction, the trolley 40 can also be temporarily stopped by manual operation by the operator P.

[0029] In step S6, when the unmanned trolley 40 resumes movement, operator P sends a pre-defined signal to the fixed-point cameras 21-26 and the trolley camera 43. Similarly to step S1, the central control unit 60 detects the signal from operator P through image processing, thereby receiving the operation to resume the unmanned movement. Upon receiving the operation to resume the unmanned trolley 40's movement, the central control unit 60 resets the trolley 40 to the operating mode. Thus, the trolley control unit 47 resumes unmanned control. Alternatively, when the trolley 40 resumes unmanned movement, operator P can also send signals to the fixed-point cameras 21-26 and the trolley camera 43 to cause the trolley 40 to perform desired actions.

[0030] In step S7, after all the work processes for the first work object W1 are completed, operator P moves to the standby position corresponding to the work of the second work object W2, which has been transported to the (N+1)th work station on the work line L, in order to begin the work. The central control unit 60 detects that operator P has moved to the standby position corresponding to the work of the second work object W2. Upon detecting that operator P has moved to the standby position corresponding to the work of the second work object W2, the central control unit 60 sets the trolley 40 to the operation mode. Furthermore, the central control unit 60 extracts a segmented image of the trolley 40 from the second learning dataset D2, which is associated with the process identification information indicating the first executed work process and the object identification information indicating the second work object W2. Based on the position and posture of the trolley 40 in the extracted segmented image, the central control unit 60 generates a control signal to move the trolley 40 to the initial position corresponding to the work of the second work object W2, and sends the generated control signal to the trolley 40. The trolley control device 47 uses control signals received from the central control device 60 to control the actuator assembly, thereby moving the trolley 40 to an initial position corresponding to the operation of the second work object W2. The operator P uses the trolley 40 to begin the operation on the second work object W2.

[0031] According to the above embodiment, in order to analyze the movements and configurations of multiple objects existing in work area A, such as trolley 40, tools, components, operator P, and work objects W1 and W2, the central control device 60 can acquire captured images from fixed-point cameras 21-26 and trolley camera 43. Furthermore, in order to analyze the work conditions in work area A, such as the configuration of trolley 40, work objects W1 and W2, the state of tools and components, and the movements of operator P, the central control device 60 can use machine learning to acquire images representing the positions and postures of the objects represented by the captured images. Moreover, the central control device 60 can use the acquired images to determine which of multiple work processes the work represented by the captured images is. Furthermore, the central control device 60 can move trolley 40 to a position corresponding to the work process determined as the work represented by the captured images. In other words, the central control device 60 can analyze the work conditions in work area A according to a time sequence based on the movements and configurations of multiple objects existing in work area A. Therefore, the central control device 60 can automatically move the trolley 40 to the appropriate position when needed, based on the work situation, or predict the next action of movable objects such as the worker P in the work site A and move the trolley 40 in advance.

[0032] Furthermore, according to the above embodiment, the central control device 60 can use key point images representing the musculoskeletal structure of the operator P to determine which of several work processes the captured image represents, and then move the trolley 40 accordingly. That is, the central control device 60 can move the trolley 40 based on the operator P's movements. This reduces the possibility of the operator P being idle due to the trolley 40 not being positioned appropriately at the required time, the operator P being required to move in accordance with the trolley 40's configuration, or interference between the operator P and the trolley 40. Moreover, during operation, the position of the trolley 40 can be successively corrected in conjunction with the operator P's movements. Therefore, for example, if the work objects W1 and W2 are large and the operator P moves from front to back during operation, the possibility of the operator P's hand not reaching the trolley 40 can be reduced. Furthermore, if the operator P represented in the captured image is a beginner, the operation system 1 can also move the trolley 40 to a position corresponding to the movements of an experienced operator. In this way, System 1 enables beginners to easily master the standard movements of experienced users.

[0033] Furthermore, according to the above embodiment, the central control device 60 can move the trolley 40 to a position corresponding to the work procedure. That is, the central control device 60 can adjust the distance between the operator P and the trolley 40 for each work procedure.

[0034] Furthermore, according to the above embodiment, the central control device 60 can refer to a second learning dataset D2 that establishes a correlation between the work objects W1 and W2 and the segmented image representing the trolley 40 positioned at the initial position corresponding to the work objects W1 and W2. Therefore, after all the work processes for the first work object W1 are completed, the central control device 60 can automatically move the trolley 40 to the initial position corresponding to the work for the second work object W2.

[0035] Furthermore, according to the above embodiment, the operating system 1 can determine the position of the trolley 40 using images captured from the fixed-point cameras 21-26 and images captured from the trolley camera 43. Thus, even when the trolley 40 is surrounded by the operator P, information can be supplemented by images captured from the fixed-point cameras 21-26. Therefore, the possibility of difficulty in determining the position of the trolley 40 can be reduced. In addition, since the images captured from the fixed-point cameras 21-26 are combined with the images captured from the trolley camera 43, the situation where the greater the distance between the fixed-point cameras 21-26 and the trolley 40, the lower the accuracy of estimating the position of the trolley 40 can be avoided. Moreover, by synchronizing the images captured from the fixed-point cameras 21-26 with the images captured from the trolley camera 43, the position of the trolley 40 can be determined not only based on a fixed object but also based on a movable object.

[0036] Furthermore, according to the above embodiment, the work system 1 can use the captured images output from the trolley camera 43 to determine the state of the tools and components within the housing 41. In this way, images captured by the trolley camera 43 from a location closer to the fixed-point cameras 21-26 are available. Therefore, the possibility of difficulty in determining the state of the tools and components in the captured images due to the operator P overlapping the tools and components is reduced.

[0037] Furthermore, according to the above embodiment, the work system 1 can determine the actions of the operator P using the images captured from the fixed-point cameras 21-26 and the images captured from the trolley camera 43. This reduces the possibility that the actions of the operator P may be difficult to determine due to other objects overlapping the operator P in the captured images, or because a part of the operator P's body is outside the detection range of the fixed-point cameras 21-26 and the trolley camera 43.

[0038] Furthermore, according to the above embodiment, the work is performed by separating the selection and assembly of components in the SPS (Separate Parts Processing System). In this case, since the trolley 40, which houses heavy tools and components, is repeatedly moved between the preparation area R and the work line L, the worker P may sometimes experience stress on their waist, legs, and wrists. In contrast, in the above embodiment, the trolley 40 can be moved unmanned. This reduces the burden on the worker P caused by physically operating the trolley 40.

[0039] Furthermore, in the above embodiment, the central control device 60 uses fixed-point cameras 21-26 and trolley camera 43 to track objects present in the work area A, thus automatically determining the progress of the work. This avoids requiring the operator P to perform additional tasks beyond operating the trolley 40. Therefore, it prevents an increase in work time due to additional tasks beyond operating the trolley 40.

[0040] B. Other implementation methods: (B1) The work objects W1 and W2 can be anything other than vehicles. The work objects W1 and W2 can be mobile bodies other than vehicles, or objects other than mobile bodies. Furthermore, if the work objects W1 and W2 are mobile bodies, the mobile bodies can also be moved autonomously. In this case, the mobile bodies are transported along the work line L using autonomous driving-based movement.

[0041] (B2) Workplace A may also be any other place besides the factory that manufactures work objects W1 and W2. Workplace A may be a factory that repairs work objects W1 and W2 after they have been shipped, a factory that inspects work objects W1 and W2 after they have been shipped, or a place other than the factory.

[0042] (B3) The trolley 40 can also be moved autonomously. In this case, the function of the central control device 60 is realized by the trolley control device 47.

[0043] (B4) Each of the multiple training images can also be further associated with category information representing the category of the work objects W1 and W2 as supplementary information. As in this embodiment, when the work objects W1 and W2 are vehicles, the category information may be, for example, information representing vehicle specifications determined by overall length, width, and height, or vehicle model information classified according to the vehicle's appearance. The category information may also be information representing factors that affect the work sequence, such as the drive method (electric motor, engine, etc.), the type and location of equipment such as a steering wheel, etc. In this way, the central control device 60 can move the trolley 40 to a position corresponding to the category of the work objects W1 and W2.

[0044] (B5) The training images representing worker P among the multiple training images can be further associated with worker information related to worker P as supplementary information. Worker information may include, for example, worker identification information identifying multiple workers P, physical information representing the physique of worker P, and characteristic information representing the characteristics of worker P such as dominant hand. In this manner, the central control device 60 can move the trolley 40 to a position corresponding to the physique and characteristics of worker P.

[0045] (B6) Machine learning models M1 and M2 can also be updated by relearning after the task is completed. This improves the accuracy of machine learning models M1 and M2. Therefore, the central control unit 60 can move the trolley 40 to a more suitable position when needed, based on the task status.

[0046] (B7) The central control device 60 can also receive operations from the operator P on the trolley 40 by recognizing the operation of buttons set on the trolley 40, the work site A, and the buttons displayed on the monitor mounted on the trolley 40.

[0047] (B8) At least a portion of the tasks performed in the preparation area R can be performed by a robot, or by a joint operation between the operator P and the robot. Alternatively, tasks can be performed using methods other than SPS.

[0048] (B9) The judgment unit 612 can also obtain process identification information indicating which of a plurality of work processes the work represented by the captured image is by inputting the captured image into the trained third machine learning model, and determine the work process determined by the process identification information as the work process represented by the captured image. The third machine learning model is a machine learning model that outputs process identification information representing the work process represented by the captured image when the captured image is input. As the third machine learning model, for example, a CNN learned by teacher training using a third learning dataset can be used. The third learning dataset includes various information contained in the first learning dataset D1 and the second learning dataset D2. The third learning dataset, for example, has at least a plurality of training images including at least one of the following objects: trolley 40, tool, component, operator P, and work object W1, W2, as well as process identification information as the label of the correct answer. If this is the case, the judgment unit 612 can determine the work process without acquiring the captured image.

[0049] This disclosure is not limited to the embodiments described above, and can be implemented in various structures without departing from its spirit. For example, in order to solve some or all of the above-described technical problems, or to achieve some or all of the above-described effects, the technical features of the embodiments corresponding to the technical features in the various methods described in the Summary of the Invention section can be appropriately replaced or combined. In addition, any technical feature that is not described as an essential technical feature in this specification can be appropriately deleted. Explanation of reference numerals in the attached figures

[0050] 1…Working system, 21~26…Fixed-point cameras, 40…Trolley, 41…Housing section, 43…Trolley camera, 47…Trolley control device, 49…Trolley body, 60…Central control device, 601…Processor, 602…Memory, 603…Input / output interface, 604…Bus, 605…Communication device, 611…Acquisition unit, 612…Judgment unit, 613…Motion control unit, A…Working area, D1…First learning dataset, D2…Second learning dataset, L…Working line, M1…First machine learning model, M2…Second machine learning model, P…Operator, PG…Program, R…Preparation area, W1…First work object, W2…Second work object.

Claims

1. An operating system, comprising: One or more stationary cameras provide an overhead view of the shooting location; One or more trolleys are capable of moving autonomously and include: a housing for accommodating tools and components for assembling into the work object; and one or more trolley cameras for capturing images of the trolleys and their surrounding area; and The control device controls the movement of the trolley and includes: an acquisition unit that acquires a captured image output from at least one of the fixed-point camera and the trolley camera; a judgment unit that, by inputting the captured image into a learned machine learning model, determines which of a plurality of work processes the work represented by the captured image is based on the position and posture of the object represented by the captured image in the image; and a motion control unit that moves the trolley to a position corresponding to the work process determined by the judgment unit.

Images