Training device, image collection device, and control device

Abstract

A training device (2) according to the present disclosure comprises: a storage unit that stores a plurality of teacher images including a region of an object (W) captured by an imaging unit (4); and a training unit that, by using the plurality of teacher images, trains a learning model for detecting the region of the object from the images. The plurality of teacher images used for training the learning model include an image focused on the object (W) and an image not focused on the object (W).

Description

Learning device, image collection device, and control device

[0001] The present disclosure relates to a learning device, an image collection device, and a control device.

[0002] In the task of depalletizing, in which a robot removes stacked boxes one by one, box detection technology is one important technique that can improve work efficiency. One box detection technique involves detecting box regions by applying threshold processing to captured images taken by a camera. However, this technique may not be able to detect box regions due to reasons such as the background color of the boxes being similar to the color of the boxes themselves, boxes being close to each other, or tape or other material being attached to the box surfaces. A technique using learning is known to address such cases (see, for example, Patent Literature 1). For example, in machine learning using teacher images, a learning model is trained to detect box regions from images using teacher images captured in advance. By inputting captured images into a trained model that has completed training, box regions contained in the captured images can be detected. In such machine learning using teacher images, the detection accuracy of the trained model varies significantly depending on the number and quality of the teacher images.

[0003] Japanese Patent Application Publication No. 11-272845

[0004] There is a need for a technology that uses machine learning to improve the accuracy of detecting an object region from a captured image.

[0005] The learning device of the present disclosure includes a memory unit that stores multiple teacher images that include the area of ​​an object captured by an imaging unit, and a learning unit that uses the multiple teacher images to learn a learning model for detecting the area of ​​the object from an image.The multiple teacher images used to learn the learning model include images that are not in focus with the object.

[0006] FIG. 1 is a diagram showing a learning system including a learning device according to a first embodiment. FIG. 2 is a hardware configuration diagram of the learning device of FIG. 1. FIG. 3 is a functional configuration diagram of the learning device of FIG. 1. FIG. 4 is a diagram showing an example of a teacher image management table stored in the storage unit of FIG. 3. FIG. 5 is a diagram showing an example of a trained model management table stored in the storage unit of FIG. 3. FIG. 6 is a flowchart showing an example of an image collection process procedure performed by the learning device of FIG. 1. FIG. 7 is a flowchart showing an example of a learning process procedure performed by the learning device of FIG. 1. FIG. 8 is a diagram showing a robot system including a control device according to a second embodiment. FIG. 9 is a hardware configuration diagram of the control device of FIG. 8. FIG. 10 is a functional configuration diagram of the control device of FIG. 8. FIG. 11 is a flowchart showing an example of a robot control procedure performed by the control device of FIG. 8.

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the accompanying drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0008] A learning device according to a first embodiment will be described below with reference to Figures 1 to 7. The learning device according to the first embodiment (hereinafter simply referred to as the learning device) is an information processing device having an image collection function that collects multiple teacher images by controlling a camera, and a learning function that uses the multiple teacher images to learn a learning model for detecting box regions from images. One feature of the learning device is that it collects not only teacher images in focus on the box, but also teacher images that are out of focus on the box, and uses these images to train the learning model.

[0009] In this embodiment, the terms are defined as follows: Box: A box is an example of an object. The object is not limited to a box, and can be any object that can be detected by image detection. Camera: An example of an imaging unit. In the first embodiment, the camera is a device that can change the focus, focal length, and aperture value based on a control signal input from outside.

[0010] Focus: "In focus" means that the box appears clearly in the image, while "out of focus" means that the box appears blurry rather than clear in the image. Optically, "in focus" means that the distance (defocus amount) from the image-forming position of the box to the imaging position is zero or close, while "out of focus" means that the distance from the image-forming position of the box to the imaging position is far. Therefore, the focus can be changed by changing the distance from the image-forming position of the box to the imaging position. For example, the focus can be changed by moving the imaging position along the optical axis. The in-focus range may be defined as appropriate. For example, a defocus amount of zero, i.e., the imaging position coincides with the image-forming position, may be considered "in focus," or a defocus amount less than a predetermined distance may be considered "in focus."

[0011] FIG. 1 is a diagram showing the configuration of a learning system including a learning device according to the first embodiment. As shown in FIG. 1, the learning system 1 is configured by connecting a camera 4 and an information processing terminal 5 to a learning device 2 that functions as the system's central processing unit. The learning device 2 is a computer device that has both a learning function for learning a learning model for detecting a box region from an image using multiple teacher images captured by the camera 4, and an image collection function for controlling the camera 4 to collect multiple teacher images, and is realized by a PC or the like. Note that the concept of the learning device may include the camera and an information processing device that has both the learning function and the image collection function.

[0012] The camera 4 is connected to the learning device 2 via a cable so that data communication is possible. The camera 4 is fixed in a position overlooking the stacked boxes W, and sets imaging conditions such as focus, aperture value, and focal length in accordance with control signals from the learning device to capture images of the boxes W. The captured image by the camera 4 is input to the learning device 2 as a teacher image. The camera 4 may also be attached to the hand of a robot positioned close to the stacked boxes W.

[0013] The information processing terminal 5 is connected to the learning device 2 via a network 3 such as the Internet so that data can be communicated therewith. The information processing terminal 5 is a terminal that functions as an input device for inputting user operations to the learning device 2, and is realized by a smartphone, tablet, or the like.

[0014] Note that the process of collecting teacher images and the process of learning a learning model are independent processes, and therefore do not necessarily have to be realized in a single system. For example, an image collection device 2' having only an image collection function and a learning device 2 having only a learning function may each be configured as independent devices.

[0015] Furthermore, the function of information processing terminal 5 for inputting user operations to learning device 2 may be implemented in learning device 2. That is, learning device 2 may have, as functions necessary for accepting user operations, a display device that displays a screen created by learning device 2 and an operation device that accepts user operations on the screen displayed on the display device.

[0016] FIG. 2 is a hardware configuration diagram of the learning device 2 according to the first embodiment. As shown in FIG. 2, the learning device 2 is configured by connecting a RAM 12, a ROM 13, a storage device 14, a camera I / F 15, and a communication device 16 to a processor 11 via a data / control bus 10. The processor 11 is implemented by a CPU, a GPU, etc. The RAM 12 functions as the processor 11's main memory, work area, etc. The ROM 13 stores a BIOS, an OS, etc. The storage device 14 stores a learning program. Various data, such as the learning program, stored in the storage device 14 may be recorded on a removable medium (non-transitory storage medium) such as a USB and distributed to a user, or may be distributed by downloading to the learning device 2 via a network 3. A camera 4 is connected to the camera I / F 15. The camera 4 sets imaging conditions and executes imaging processing under the control of the processor 11. The communication device 16 is realized by a communication module that complies with any communication standard, and transmits and receives data to and from the information processing terminal 5 under the control of the processor 11 .

[0017] 3 is a functional configuration diagram of the learning device 2. When the learning program loaded from the storage device 14 to the RAM 12 is executed by the processor 11, the learning device 2 functions as a transmitter 21, an operation information input unit 22, a teacher image input unit 23, a memory unit 24, a screen creation unit 25, an imaging condition setting unit 26, a camera control unit 27, a learning condition setting unit 28, and a learning unit 29.

[0018] The transmission unit 21 transmits data of various screens related to the learning program created by the screen creation unit 25 to the information processing terminal. The operation information input unit 22 inputs user operations via the information processing terminal. Specifically, the operation information input unit 22 functions to input object information, layout information, image collection conditions, learning conditions, imaging start instructions, etc. The teacher image input unit 23 inputs data of teacher images captured by the camera 4. The input teacher image data is stored in the storage unit 24. Furthermore, the object information, layout information, and imaging conditions related to the teacher images stored in the storage unit 24 are registered in a teacher image management table stored in the storage unit 24.

[0019] The storage unit 24 stores various information related to the processing of the learning program. Specifically, the storage unit 24 stores data of teacher images captured by the camera 4, data of trained models trained by the learning unit 29, a teacher image management table that manages information about the teacher images, and a trained model management table that manages information about the trained models. Details of these management tables will be described later.

[0020] The screen creation unit 25 creates various screens related to the learning program in accordance with a predetermined format. The various screens include an input screen for inputting object information and layout information, a setting screen for image collection conditions, a setting screen for learning conditions, an instruction screen for instructing the start of imaging, etc.

[0021] The imaging condition setting unit 26 sets the number of teacher images to be captured and the imaging conditions for each teacher image to be captured based on the image collection conditions input via the operation information input unit 22. The image collection conditions include a focus setting range and pitch, an aperture setting range and pitch, and a focal length setting range and pitch. The imaging condition setting unit 26 sets a plurality of imaging conditions that differ in at least one of focus, aperture, and focal length based on the image collection conditions. For example, when the imaging condition setting unit 26 sets three focus patterns, two aperture value patterns, and four focal length patterns based on the image collection conditions, 24, which is the total number of these patterns, is set as the number of teacher images to be captured, and 24 imaging condition patterns are set.

[0022] When an instruction to start imaging is input via the operation information input unit 22, the camera control unit 27 controls the camera 4 to acquire a teacher image under the imaging conditions set by the imaging condition setting unit 26.

[0023] The learning condition setting unit 28 sets the learning conditions input via the operation information input unit 22. The learning conditions are conditions for narrowing down the teacher images to be used for learning the learning model from the plurality of teacher images stored in the storage unit 24. The learning conditions include object conditions, layout conditions, and imaging condition range (imaging condition set). The learning conditions will be described in detail later.

[0024] The learning unit 29 uses a plurality of teacher images that satisfy the learning conditions set by the learning condition setting unit 28 to learn a learning model for detecting box areas from images. For example, the learning unit 29 learns the learning model through machine learning using the teacher images and the box areas in the teacher images as teacher data. As a result, when an image captured by the camera 4 is input, a trained model that detects box areas included in the image is created. The trained model created by the learning unit 29 is stored in the memory unit 24, and the learning conditions of the trained model stored in the memory unit 24 are registered in a trained model management table. Note that this does not deny the possibility of training the learning model using all teacher images; if the learning condition is all teacher images, the learning unit 29 creates a trained model trained using all teacher images.

[0025] The teacher image management table stored in the storage unit 24 will be described below with reference to Fig. 4. Fig. 4 shows an example of the teacher image management table stored in the storage unit 24. As shown in Fig. 4, in the teacher image management table, an "image number" that uniquely identifies a teacher image is associated with a "file name," "object information," "layout information," and "imaging conditions." The object information, layout information, and imaging conditions are information used as learning conditions for extracting a teacher image to be used for learning from a plurality of teacher images.

[0026] The object information is information about the object being imaged. For example, the object information includes information about the type of object, such as a box or a pallet, detailed information about the object, such as its shape, dimensions, color, and material, and supplemental information, such as whether a label, tape, or the like is attached to the surface of the object.

[0027] The layout information indicates the layout in which the image was captured. For example, the layout information includes an imaging distance, stacking information, and environmental information. The imaging distance indicates the physical distance from the camera 4 to the object. The stacking information indicates how the boxes W were stacked. For example, the stacking information includes 3 rows and 3 columns, 4 rows and 4 columns, etc. 3 rows and 3 columns indicates that one row is made up of 3 boxes vertically and 3 boxes horizontally, for a total of 9 boxes W. The environmental information indicates the surrounding environment at the time of capturing the image. For example, the environmental information includes information that may affect the detection of the object region from the teacher image, such as the color of the lighting, the background color of the boxes W included in the field of view of the camera 4 (the color of the floor on which the boxes W are stacked), and whether or not structures and shadows of structures are reflected within the field of view of the camera 4.

[0028] The imaging conditions represent the optical setting conditions of the camera 4 at the time of imaging. For example, the imaging conditions include focus (defocus amount), aperture value, and focal length.

[0029] The trained model management table stored in the storage unit 24 will be described below with reference to FIG. 5. FIG. 5 shows an example of the trained model management table stored in the storage unit 24. As shown in FIG. 5, in the trained model management table, a "model name" and a "learning condition" are associated with a "model number" that uniquely identifies a trained model. The model name represents the name of the trained model. The learning condition is a condition used to narrow down the teacher images used for training the trained model. The learning condition includes an "object condition," a "layout condition," and an "imaging condition range."

[0030] The object condition is a condition for narrowing down the teacher images to be learned based on the object information. The object condition is determined from the object information managed in the teacher image management table. For example, the object condition narrows down the teacher images to be learned to teacher images associated with the object type "box" and shape "square."

[0031] The layout conditions are conditions for narrowing down the teacher images to be learned using layout information. The layout conditions include an imaging distance condition, a stacking condition, and an environmental condition, and the imaging distance condition represents the range of imaging distance. The stacking condition is selected from the stacking information managed in the teacher image management table. The environmental condition is selected from the environmental information managed in the teacher image management table. For example, the layout conditions narrow down the teacher images to be learned to teacher images associated with an imaging distance condition of "200 cm to 300 cm," stacking conditions of "3 rows and 3 columns" and "4 rows and 4 columns," and environmental conditions of "lighting color 'white' and floor color 'green'."

[0032] The imaging condition range is a plurality of imaging conditions for narrowing down the teacher images to be learned. The imaging condition range includes a focus range, an aperture value range, and a focal length range. For example, the imaging condition range narrows down the teacher images to be learned to teacher images associated with a focus range (defocus amount range) of "0.0 mm to 0.2 mm," an aperture value range of "f1.4 to f2.8," and a focal length range of "10 mm to 35 mm."

[0033] The image collection process performed by the learning device 2 will be described below with reference to FIG. 6 . FIG. 6 is a flowchart illustrating an example of the image collection process performed by the learning device 2. As shown in FIG. 6 , the learning device 2 first sets object information and layout information input from the information processing terminal (S11). Next, the learning device 2 sets the number of teacher images to be captured and the imaging conditions for each teacher image based on the image collection conditions input from the information processing terminal (S12). When an instruction to start imaging is input from the information processing terminal, the learning device 2 selects one of the imaging conditions (focus, aperture value, focal length) from the set imaging conditions (S13) and transmits a control signal to the camera 4 instructing the camera 4 to start imaging together with the selected imaging condition (S14). Through the processing of step S14, the camera 4 sets the imaging conditions selected in step S13 and executes imaging processing. The learning device 2 receives data of the teacher images captured under the imaging conditions set in step S13 from the camera 4 (S15), stores the data (S16), assigns an image number to the stored teacher image, and registers the object information, layout information, and imaging conditions in the teacher image management table (S17). The processes of steps S13 to S17 are repeatedly executed until imaging of all teacher images to be captured is completed (S18; No). Upon completion of imaging of all teacher images to be captured, the image collection process ends (S18; Yes). According to the image collection process of steps S11 to S18, the learning device 2 can acquire multiple teacher images, including teacher images in focus on the box W and teacher images that are out of focus on the box W, by controlling the camera 4 based on the input image collection conditions.

[0034] The learning process performed by the learning device 2 will be described below with reference to FIG. 7 . FIG. 7 is a flowchart illustrating an example of the learning process performed by the learning device 2. As shown in FIG. 7 , the learning device 2 first sets learning conditions input from an information processing terminal (S21) and selects a teacher image that satisfies the learning conditions from among multiple teacher images (S22). Next, the learning device 2 trains a learning model using the selected teacher image (S23), stores the trained model (S24), and registers the learning conditions of the trained model in a trained model management table (S25). According to the learning process involving steps S21 to S25, the learning device 2 can create a trained model using multiple teacher images, including a teacher image that is in focus on the box W and a teacher image that is out of focus on the box W.

[0035] 1 to 7 , it is possible to collect a plurality of teacher images including teacher images that are in focus on the box W and teacher images that are out of focus on the box W, and to create a trained model that detects box regions from images using a plurality of teacher images that are in focus on the box W and teacher images that are out of focus on the box W. A trained model that is trained intentionally using teacher images that are out of focus on the box W has the following effects.

[0036] That is, in a system in which a robot removes stacked boxes W, as the removal task of the boxes W progresses and the number of stacked boxes W decreases, the distance of the camera 4 to the box W to be removed changes, and the camera 4 may not be in focus on the box W to be removed. Specifically, for example, when removing boxes W stacked nine high, if the camera 4 is focused on the box W stacked on the ninth level, the focus on the box W may begin to shift as the number of levels decreases, and the camera 4 may not be in focus on the box W stacked on the first level. In such a case, by using a trained model that is trained using intentionally out-of-focus teacher images, the possibility of detecting a box region from a captured image in which the box W is out of focus can be improved compared to using a trained model trained only using in-focus teacher images. In other words, the accuracy of detecting a box region from a captured image in which the box W is out of focus can be improved.

[0037] Furthermore, the learning device 2 according to the first embodiment can create a trained model trained using teacher images narrowed down from multiple teacher images based on learning conditions (object conditions, layout conditions, and imaging condition range). For example, a trained model trained using teacher images narrowed down under layout conditions similar to the actual layout or imaging conditions similar to the imaging conditions actually used can shorten the training time while maintaining the accuracy of box region detection compared to a trained model trained using all teacher images. Furthermore, teacher images captured under layouts significantly different from the actual layout or imaging conditions significantly different from the imaging conditions actually used may result in noise, reducing the accuracy of box region detection. In such cases, a trained model trained using teacher images narrowed down under layout conditions similar to the actual layout or imaging conditions similar to the imaging conditions actually used can improve the accuracy of box region detection compared to a trained model trained using all teacher images.

[0038] A control device according to the second embodiment will be described below with reference to Figures 8 to 11. Note that components having the same functions as those in the first embodiment are given the same reference numerals, and detailed descriptions thereof will be omitted.

[0039] The control device (hereinafter simply referred to as the learning device) according to the second embodiment is an information processing device having a control function that detects a box region from an image captured by a camera using a trained model that has been machine-learned using multiple training images as training data, and controls a robot based on the detection results. One feature of the control device is that, in the process of causing the robot to perform an operation to pick up a box, the control device detects a box region from an image using a trained model that has been trained using multiple training images that include both training images in which the box is in focus and training images in which the box is out of focus.

[0040] Fig. 8 is a diagram showing the configuration of a robot system including a control device according to the second embodiment. As shown in Fig. 8, a robot system 6 is configured by connecting a camera 4, an information processing terminal 5, a robot 8, and a three-dimensional sensor 9 to a control device 7 that functions as the core of the system.

[0041] The robot 8 is connected to the control device 7 via a cable so as to be able to communicate data with the control device 7. The robot 8 is placed in a position close to the stacked boxes W, and performs an operation of picking up the boxes W in accordance with a control signal from the control device 7.

[0042] The three-dimensional sensor 9 is connected to the control device 7 via a cable so as to be able to communicate data. The three-dimensional sensor 9 is fixed in a position overlooking the stacked boxes W, and transmits data relating to the distance to the surface of the box W to be picked up to the control device 7 in accordance with a control signal from the control device 7.

[0043] FIG. 9 is a hardware configuration diagram of a control device 7 according to the second embodiment. As shown in FIG. 9 , the control device 7 is configured by connecting a RAM 12, a ROM 13, a storage device 14, a camera I / F 15, a communication device 16, a robot I / F 17, and a sensor I / F 18 to a processor 11 via a data / control bus 10. The storage device 14 stores a control program for implementing a function for controlling the robot 8. Various data, such as the control program, stored in the storage device 14 may be recorded on a removable medium (non-transitory storage medium) such as a USB and distributed to a user, or may be distributed by being downloaded to the control device 7 via the network 3. The robot 8 is connected to the robot I / F 17. The robot 8 operates under the control of the processor 11. The three-dimensional sensor 9 is connected to the sensor I / F 18. The three-dimensional sensor 9 measures the distance from a predetermined reference position of the three-dimensional sensor 9 to a box W (a measurement target) under the control of the processor 11.

[0044] 10 is a functional configuration diagram of the control device 7. When the processor 11 executes a control program loaded from the storage device 14 to the RAM 12, the control device 7 functions as a transmission unit 71, an operation information input unit 72, a captured image input unit 73, a distance input unit 74, a memory unit 75, a screen creation unit 76, an imaging condition setting unit 77, a camera control unit 78, a trained model setting unit 79, a box detection unit 80, a three-dimensional sensor control unit 81, and a robot control unit 82.

[0045] The transmission unit 71 transmits data of various screens related to the control program created by the screen creation unit 76 to the information processing terminal 5. The operation information input unit 72 inputs user operations via the information processing terminal 5. Specifically, the operation information input unit 72 functions to input setting conditions for the trained model, image capture conditions, operation start instructions, and the like. The captured image input unit 73 inputs data of the captured image captured by the camera 4. The distance input unit 74 inputs data related to the distance to the box W to be removed, measured by the three-dimensional sensor 9. The memory unit 75 stores various information related to the processing of the control program. Specifically, the memory unit 75 stores data of multiple trained models in which at least some of the teacher images used for training are different, and a trained model management table that manages information about the trained models. The trained model management table is the same as that in the first embodiment, and therefore description thereof will be omitted.

[0046] The screen creation unit 76 creates various screens related to the control program according to a predetermined format. The various screens include an imaging condition setting screen, a learned model setting screen, and an instruction screen for instructing the start of the removal work. The imaging condition setting unit 77 sets the imaging conditions input via the operation information input unit 72. The imaging conditions include focus, aperture value, and focal length. When an instruction to start work is input via the operation information input unit 72, the camera control unit 78 controls the camera 4 to acquire an image under the imaging conditions set by the imaging condition setting unit 77.

[0047] The trained model setting unit 79 selects one trained model from the plurality of trained models based on the trained model setting conditions input via the operation information input unit 72, and sets the selected trained model to the box detection unit 80. Note that the trained model setting unit 79 may also narrow down the plurality of trained models based on the trained model setting conditions, and set the trained model selected by the user from the narrowed down at least two trained models to the box detection unit 80.

[0048] The box detection unit 80 uses the trained model set by the trained model setting unit 79 to recognize a box area from the image captured by the camera 4 and detect the position of the recognized box area. The box detection unit 80 sets the detected box W as the box to be removed. The three-dimensional sensor control unit 81 controls the three-dimensional sensor 9 to acquire the distance from the three-dimensional sensor 9 to the box W to be removed. The robot control unit 82 controls the robot 8 to remove the box W to be removed, based on the position of the box area detected by the box detection unit 80 and the distance detected by the three-dimensional sensor 9.

[0049] 11, a robot control process performed by the control device 7 to cause the robot 8 to pick up the box W will be described. FIG. 11 is a flowchart showing an example of a robot control process performed by the control device 7.

[0050] As shown in FIG. 11 , the control device 7 first accepts the setting conditions (object information, layout information, and imaging conditions) of the trained model input from the information processing terminal 5 (S31) and selects one trained model from the multiple trained models based on the accepted setting conditions (S32). Next, in response to a work start instruction input from the information processing terminal 5, the control device 7 transmits an imaging instruction to the camera 4 along with the imaging conditions input from the information processing terminal 5 (S33). Through the processing of step S33, the imaging conditions set by the user are set to the camera 4, and the imaging processing is executed. The control device 7 receives captured image data from the camera 4 (S34) and executes a box region detection process from the captured image using the trained model (S35). If the box region cannot be detected (S36; No) but stacked boxes W remain (S41; Yes), the control device 7 changes the trained model (S42), returns to the processing of step S33, and executes the box W detection process again using the changed trained model. Whether or not any stacked boxes W remain can be determined based on the image captured by the camera 4, the three-dimensional measurement results by the three-dimensional sensor 9, and the like. On the other hand, when a box region is detected (S36; Yes), an instruction to measure the distance from the three-dimensional sensor 9 to the box W to be removed is sent to the three-dimensional sensor 9 (S37). Through the processing of step S37, the distance from the three-dimensional sensor 9 to the box W to be removed is input to the control device 7. Based on the position of the box region detected in step S35 and the distance from the three-dimensional sensor 9 to the box W to be removed acquired through the processing of step S37, the control device 7 sends an operation instruction to the robot 8 to remove the box W to be removed (S38). Through the processing of step S38, the robot 8 removes the box W to be removed. The control device 7 repeatedly executes the processing of steps S37 and S38 until all of the boxes W detected in step S35 have been removed (S39; No). The control device 7 repeatedly executes the processes of steps S33 to S39 until all stacked boxes W have been removed (S40; Yes), and terminates control of the robot 8 when all stacked boxes W have been removed (S40; No).

[0051] The control device 7 according to the second embodiment described above can initialize a trained model in accordance with user instructions, execute a box W detection process using the initialized trained model, and control the robot 8 to retrieve the detected box W. The trained model is trained using multiple training images, including training images in which the box W is in focus and training images in which the box W is out of focus. As described in the first embodiment, a trained model trained using out-of-focus training images can detect a box region even from an out-of-focus image. Therefore, even if an out-of-focus image of the target box W is captured during actual work, the accuracy of detecting the box W from an out-of-focus image can be improved compared to a trained model trained using only in-focus training images. This can prevent the robot 8 from stopping due to an inability to detect the box W during the robot 8's task of retrieving the box W.

[0052] Furthermore, according to the control device 7 of the second embodiment, if the box W cannot be detected using a trained model, the detection process for the box W can be repeatedly executed while switching between multiple pre-stored trained models. The multiple pre-stored trained models have also been trained using teacher images in which the box W is not in focus, and the learning conditions are different from each other, so that the possibility of detecting the box W can be improved by switching between trained models.

[0053] In this embodiment, camera 4 has been described as a device that can automatically change imaging conditions such as focus, focal length, and aperture value through internal processing of camera 4 based on control signals input from the outside, but the imaging conditions such as focus, focal length, and aperture value of camera 4 may also be set by an operator manually operating camera 4. In this case, instead of step S14 in the image collection processing procedure described with reference to FIG. 6 , learning device 2 may display the imaging conditions (focus, aperture value, focal length) of the teacher image to be collected next.

[0054] In addition, in this embodiment, the focus, focal length, and aperture value of the camera 4 are each described as being changed, but it is sufficient that at least the focus can be changed, and the camera 4 is also sufficient that at least the focus can be adjusted.

[0055] The following supplementary notes are further disclosed regarding this embodiment and its modified examples. (Supplementary Note 1) The learning device 2 includes a storage unit 24 that stores a plurality of teacher images including an area of ​​an object captured by the imaging unit 4, and a learning unit 29 that uses the plurality of teacher images to learn a learning model for detecting the area of ​​the object from an image. The plurality of teacher images includes images in which the object is out of focus. (Supplementary Note 2) In the learning device 2 described in Supplementary Note 1, the plurality of teacher images include at least two teacher images for which the imaging unit 4 has different focal lengths. (Supplementary Note 3) In the learning device 2 described in Supplementary Note 1 or Supplementary Note 2, the plurality of teacher images include at least two teacher images for which the imaging unit 4 has different aperture values. (Supplementary Note 4) In the learning device 2 described in any of Supplements 1 to 3, the plurality of teacher images include at least two teacher images for which the physical distance from the imaging unit 4 to the object is different. (Supplementary Note 5) The learning device 2 described in any one of Supplements 1 to 4 further includes an input unit 22 for inputting learning conditions including at least one of the physical distance from the imaging unit 4 to the object, the focus of the imaging unit 4, the aperture value of the imaging unit 4, and the focal length of the imaging unit 4. The learning unit 29 trains a learning model using a teacher image from among a plurality of teacher images that satisfies the learning conditions. (Supplementary Note 6) The image collection device 2′ includes an imaging unit 4 that captures images of an object and acquires a plurality of teacher images used to train a learning model for detecting an object region from the image, and a control unit 27 that controls the imaging unit to acquire the plurality of teacher images. The control unit 27 controls the imaging unit 4 to acquire a teacher image that is in focus on the object and a teacher image that is out of focus on the object. (Supplementary Note 7) In the image collection device 2′ described in Supplementary Note 6, the control unit 27 controls the imaging unit 4 to acquire at least two teacher images with different focal lengths. (Supplementary Note 8) In the image collecting device 2' described in Supplementary Note 7, the control unit 27 controls the imaging unit 4 to acquire at least two teacher images with different aperture values.(Appendix 9) The control device 7 includes an imaging unit 4 that captures an image of an object, a detection unit 80 that detects the area of ​​the object from the image captured by the imaging unit 4 using a trained model that has been machine-learned using a plurality of training images that include images in focus on the object and images out of focus on the object as training data, and a control unit 82 that controls the robot 8 based on the position of the area of ​​the object detected by the detection unit 80.

[0056] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the invention or the concept and spirit of the present invention derived from the content of the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0057] 1...Learning system, 2...Learning device, 2'...Image collection device, 3...Network, 4...Camera, 5...Information processing terminal, 6...Robot system, 7...Control device, 8...Robot, 9...3D sensor, 10...Data / control bus, 11...Processor, 12...RAM, 13...ROM, 14...Storage device, 15...Camera I / F, 16...Communication device, 17...Robot I / F, 18...Sensor I / F, 21...Transmitter, 22...Operation information input unit, 2 3...teacher image input unit, 24...memory unit, 25...screen creation unit, 26...imaging condition setting unit, 27...camera control unit, 28...learning condition setting unit, 29...learning unit, 71...transmission unit, 72...operation information input unit, 73...captured image input unit, 74...distance input unit, 75...memory unit, 76...screen creation unit, 77...imaging condition setting unit, 78...camera control unit, 79...learned model setting unit, 80...box detection unit, 81...3D sensor control unit, 82...robot control unit.

Claims

1. A storage unit that stores multiple training images that include the region of the object captured by the imaging unit, The system comprises a learning unit that learns a learning model for detecting the region of an object from an image using the aforementioned plurality of training images, The aforementioned group of training images includes images that are out of focus on the object. Learning device.

2. The plurality of training images include at least two training images with different focal lengths of the imaging unit. The learning device according to claim 1.

3. The plurality of training images include at least two training images with different aperture values ​​of the imaging unit. The learning device according to claim 1 or 2.

4. The plurality of training images include at least two training images that are at different physical distances from the imaging unit to the object. The learning device according to claim 1 or 2.

5. The system further comprises an input unit for inputting learning conditions including at least one of the following: the physical distance from the imaging unit to the object, the focus of the imaging unit, the aperture value of the imaging unit, and the focal length of the imaging unit. The learning unit learns the learning model using the training image that satisfies the learning conditions from among the plurality of training images. The learning device according to claim 1 or 2.

6. An imaging unit that captures an image of an object and acquires multiple training images to be used for training a learning model to detect the region of the object from the image, The system comprises a control unit for controlling the imaging unit in order to collect the aforementioned plurality of training images, The control unit controls the imaging unit to acquire a training image that is in focus on the target object, and to acquire a training image that is out of focus on the target object.

7. The control unit controls the imaging unit to obtain at least two training images with different focal lengths. The image acquisition device according to claim 6.

8. The control unit controls the imaging unit to obtain at least two training images with different aperture values. The image acquisition device according to claim 6 or claim 7.

9. An imaging unit that images the target object, A detection unit detects the region of the object from an image captured by the imaging unit, using a trained model that has been machine-learned using multiple training images, including images in focus on the object and images out of focus on the object, as training data. Based on the position of the region of the object detected by the detection unit, a control unit controls the robot. A control device equipped with the following.