ROBOT SYSTEM, PROCESSING METHOD, AND PROGRAM

Abstract

This robot system comprises: a robot arm that is capable of grasping an object, the object including a packaging member having transparency and a body of matter to be packaged with the packaging member; a drive means that drives the robot arm; an identification means that identifies a type of the body of matter on the basis of image processing on an image of the object; and a change means that changes, if the identification means fails to identify the type of the substance, changes an environment in which the image of the object was taken to a different environment.

Description

[Technical Field]

[0001] The present disclosure relates to a robotic system, a processing method, and program Regarding. [Background technology]

[0002] Robots are used in various fields such as logistics. Patent Documents 1 and 2 disclose related technologies relating to a robot system that grasps an object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-176334 [Patent Document 2] Patent No. 6752615 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in an image of an object such as a product covered with a transparent packaging material, there is a possibility that light is reflected by the packaging material or surrounding objects are reflected in the image. Therefore, it may be difficult to identify the object from such an image. Even if the robots disclosed in Patent Documents 1 and 2 are used, they cannot accurately recognize the object in such a situation, and therefore cannot take appropriate action against the object.

[0005] The aspects of the present disclosure provide a robot system, a processing method, and program One of the aims is to provide [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a robotic system includes: The robot arm is equipped with a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material; a driving means for driving the robot arm; an identification means for identifying the type of object based on image processing of an image of the object; a modification means for changing the environment in which the image of the object was taken to a different environment if the identification means is unable to identify the type of object; and a photographing means for photographing the image of the object, wherein the modification means controls the photographing means to take an image of the object at an angle different from the angle of the photographing means that took the image of the object if the identification means is unable to identify the type of object. According to another aspect of the present disclosure, a robot system includes a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of an image of the object; a modification means for changing the environment in which the image of the object was taken to a different environment if the identification means is unable to identify the type of the object; and an illumination means for illuminating the object, wherein if the identification means is unable to identify the type of the object, the modification means changes the light condition to be different from the light condition irradiated on the object when the image of the object was taken, and controls the illumination means to illuminate the object at an angle different from the light angle when the image of the object was taken. According to another aspect of the present disclosure, a robot system includes a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material; a driving means for driving the robot arm; an identification means for identifying a type of the object based on image processing of an image of the object; a modification means for changing an environment from the environment in which the image of the object was taken to a different environment if the identification means is unable to identify the type of the object; and an illumination means for illuminating the object, wherein if the identification means is unable to identify the type of the object, the modification means changes the light state to a different state from the light state irradiated on the object when the image of the object was taken, and moves an object that changes the refractive index of light between the object and the illumination means. According to another aspect of the present disclosure, a robot system includes a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material, a driving means for driving the robot arm, an identification means for identifying a type of the object based on image processing of an image of the object, and a modification means for changing an environment from one in which an image of the object was captured to a different environment if the identification means is unable to identify the type of the object, and the modification means controls the driving means to change the orientation of the object if the identification means is unable to identify the type of the object. According to another aspect of the present disclosure, a robot system includes a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material, a driving means for driving the robot arm, an identification means for identifying the type of the object based on image processing of an image of the object, and a modification means for changing the environment from the environment in which the image of the object was taken to a different environment if the identification means is unable to identify the type of the object, wherein the modification means performs a first processing content if the identification means is unable to identify the type of the object, and performs a second processing content different from the first processing content if the identification means is unable to identify the type of the object based on an image of the object taken after the first processing content is performed.

[0007] According to another aspect of the present disclosure, a processing method includes: A processing method executed by a robot system having a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material, the method comprising: driving the robot arm; identifying the type of the object based on image processing of an image of the object; if the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; and, if the type of the object cannot be identified, controlling the photographing means to take the image of the object at an angle different from the angle of the photographing means used to take the image of the object.

[0008] According to another aspect of the present disclosure, a program includes: A computer included in a robot system including a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material is caused to perform the following operations: drive the robot arm; identify the type of the object based on image processing of an image of the object; if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment; and if the type of the object cannot be identified, control the imaging means so that the angle of the imaging means used to take the image of the object is different from the angle of the imaging means used to take the image of the object. According to another aspect of the present disclosure, a program causes a computer of a robot system including a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material to: drive the robot arm; identify the type of the object based on image processing of an image of the object; and, if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment; and, if the type of the object cannot be identified, change the light condition to be different from the light condition irradiated on the object when the image of the object was taken, and control a lighting means that illuminates the object at an angle different from the light angle when the image of the object was taken. According to another aspect of the present disclosure, a program causes a computer of a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object packaged by the packaging member to: drive the robot arm; identify the type of the object based on image processing of an image of the object; if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment; if the type of the object cannot be identified, change the light condition to be different from the light condition irradiated on the object when the image of the object was taken, and move an object that changes the refractive index of light between the object and a lighting means that illuminates the object. According to another aspect of the present disclosure, a program causes a computer of a robot system including a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material to: drive the robot arm; identify the type of the object based on image processing of an image of the object; if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment; and if the type of the object cannot be identified, control a driving means that drives the robot arm to change the orientation of the object. According to another aspect of the present disclosure, a program causes a computer of a robot system including a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material to: drive the robot arm; identify the type of the object based on image processing of an image of the object; if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment; if the type of the object cannot be identified, perform a first processing content; and if the type of the object cannot be identified based on an image of the object taken after performing the first processing content, perform a second processing content different from the first processing content.

[0009] According to another aspect of the present disclosure, a robot system includes a robot arm and a control means for controlling the operation of the robot arm so that the robot arm performs an action on an object based on a recognition result of an image of the object photographed by an imaging device, wherein the object is an object packaged in a transparent packaging material, and when the control means cannot recognize the object from the image, the control means controls the robot arm to change the environment in which the imaging device photographs the object. [Effects of the Invention]

[0010] According to each aspect of the present disclosure, even if an object is packaged in a packaging material that has transparency, the object can be appropriately and accurately recognized. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a robot system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a database according to the first embodiment of the present disclosure. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a processing device according to a first embodiment of the present disclosure. [Figure 4]FIG. 2 is a diagram illustrating an example of a configuration of a processing unit according to the first embodiment of the present disclosure. [Figure 5] FIG. 2 is a diagram illustrating an example of training data according to the first embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating an example of a processing flow of the robot system according to the first embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating an example of a configuration of a robot according to a first modified example of the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a data table according to a modified example of the second embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates a minimal configuration of a robotic system according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of a processing flow of a robot system having a minimum configuration. [Figure 11] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The embodiments will be described in detail below with reference to the drawings. A robot system 1 according to each embodiment of the present disclosure can change the environment in which an image of an object (e.g., a product) packaged in a transparent packaging material is photographed to an environment in which an image that identifies the object can be photographed. This change in environment includes an action of the robot 20 relative to the packaging material, such as stretching the packaging material, in the robot system 1 described below, or an action of the robot 20 that changes the state of the object, such as changing the orientation of the object. First Embodiment The robot system 1 according to the first embodiment of the present disclosure can appropriately and accurately recognize an object (e.g., a product) even if the object is packaged in a transparent packaging material. Examples of transparent packaging materials include plastic wrap, vinyl, and plastic containers. The robot system 1 identifies an object to be grasped from among multiple types of objects based on image processing of captured images. In this disclosure, grasping includes not only holding an object at the position of a robot arm by clamping the object, but also holding the object at the position of a robot arm by suction. The robot system 1 may also identify an object to be subjected to an action from among multiple types of objects based on image processing of captured images. The action does not have to be grasping as described above, and may be, for example, rotation, movement, opening, boxing, etc. The action is not limited to the above examples. The robot system 1 is used, for example, in warehouses, food factories, supermarkets, convenience stores, etc.

[0013] (Robot system configuration) 1 is a diagram illustrating an example of the configuration of a robot system 1 according to a first embodiment of the present disclosure. As shown in FIG. 1, the robot system 1 includes a transport device 10, a robot 20, an illumination device 30 (an example of illumination means), and a processing device 40.

[0014] As shown in FIG. 1, the conveying device 10 includes a conveying mechanism 101, a tray T, and a database DB. Based on the database DB, the conveying mechanism 101 places multiple types of objects (e.g., products) wrapped in transparent packaging materials on the tray T and moves the tray T to a position where the robot 20 can grasp the objects. For example, products (e.g., product A, product B, and product C) received at a warehouse, food factory, supermarket, convenience store, etc. are placed on the tray T, and information indicating the type and quantity of the received products is recorded for each tray T in the database DB. FIG. 2 is a diagram illustrating an example of the database DB according to the first embodiment of the present disclosure. In the example of the database DB illustrated in FIG. 2, three types of products, namely product A, product B, and product C, are placed on the tray T1, and it is shown that there is one quantity of product A, two quantities of product B, and three quantities of product C. When the robot 20 is to grasp product A as an object, the conveying mechanism 101 identifies the tray T1 associated with product A in the database DB. Then, the transport mechanism 101 moves the tray T1 to a position where the robot 20 can grasp it.

[0015] As shown in FIG. 1, the robot 20 includes a robot arm 201, a camera 202 (an example of a camera means), and a drive mechanism 203 (an example of a drive means). The robot arm 201 grasps an object in accordance with the operation of the drive mechanism 203. The camera 202 takes an image of the object on the tray T. For example, the camera 202 takes an image of multiple types of objects (e.g., products) packaged in a transparent packaging material, or a barcode or tag attached to the object that indicates the identity of the object. Examples of the camera 202 include a camera and a video camera. The drive mechanism 203 operates the robot arm 201 under the control of the processing device 40. The lighting device 30 illuminates the object placed on the tray T.

[0016] 3 is a diagram illustrating an example of the configuration of the processing device 40 according to the first embodiment of the present disclosure. As shown in FIG. 3, the processing device 40 includes an acquisition unit 401 and a processing unit 402. The acquisition unit 401 acquires an image of an object captured by the imaging device 202.

[0017] Fig. 4 is a diagram illustrating an example of the configuration of the processing unit 402 according to the first embodiment of the present disclosure. As shown in Fig. 4, the processing unit 402 includes a specifying unit 4021 (an example of a specifying means), a control unit 4022 (an example of a control means), and a changing unit 4023 (an example of a changing means, an example of a control means).

[0018] The identification unit 4021 identifies the type of object based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object captured by the imaging device 202). For example, the identification unit 4021 compares the image of the object acquired by the acquisition unit 401 with images of each of a plurality of types of objects prepared in advance. The identification unit 4021 identifies the type of object in the image acquired by the acquisition unit 401 based on the comparison result. Alternatively, the identification unit 4021 determines that the type of object cannot be identified based on the comparison result. For example, the identification unit 4021 may identify the type of object in the image acquired by the acquisition unit 401 by applying a model created by machine learning, such as a neural network, to the image. Furthermore, for example, the identification unit 4021 may identify the object shown in the image acquired by the acquisition unit 401 as the object shown in the prepared image with the largest number of matching image portions. Furthermore, for example, the identification unit 4021 reads a barcode or tag attached to the object shown in the image acquired by the acquisition unit 401. Then, when the identification unit 4021 can identify the object by reading it, it may identify that object as the object shown in the image acquired by the acquisition unit 401. In other words, the above-described processing can also be said to be processing in which the identification unit 4021 analyzes an image and executes an operation to recognize the object shown in the image.

[0019] The control unit 4022 controls the driving mechanism 203 to grip the identified object when the identification unit 4021 can identify the type of object shown in the image acquired by the acquisition unit 401. In other words, the above-described processing can also be said to be processing in which the control unit 4022 controls the operation of the robot arm 201 based on the recognition result so that the robot arm 201 performs an operation (for example, a gripping operation) on the target object when the object is recognized from the image.

[0020] The change unit 4023 changes the environment from that in which the image was taken to a different environment when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401. Examples of reasons why the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401 include the refractive index of the transparent packaging material, reflections (light source, glare) on the surface of the transparent packaging material, and the position and orientation of the object within the transparent packaging material.

[0021] The following is an example of a process for resolving the inability of the identification unit 4021 to identify the type of object shown in the image acquired by the acquisition unit 401. For example, when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 controls the image capturing device 202 so that the angle is different from the angle of the image capturing device 202 that captured the image of the object (i.e., the target object).

[0022] Furthermore, for example, when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 changes the state of light to be different from the state of light irradiated on the object (i.e., target object) in the state in which the image of the object was captured. Specifically, when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 changes the angle of light to be different from the angle of light irradiated on the object (i.e., target object) in the state in which the image of the object was captured. Specifically, when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 moves an object that changes the refractive index of light between the object (i.e., target object) and the lighting device 30. Specifically, when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the state of the object (i.e., target object). More specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the orientation of the object (i.e., the target object). Also, more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the state of the packaging material. Even more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to suppress the bulge of the packaging material. Also, even more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to stretch the packaging material.

[0023] Furthermore, the processing of the change unit 4023 may be realized by the control unit 4022 controlling the operation of the robot arm 201. When the target object is an object wrapped in a transparent wrapping material, the above-described processing can also be said to be processing in which the control unit 4022 controls the operation of the robot arm 201 to change the environment in which the image capturing device captures the target object if the object cannot be recognized from the image.

[0024] In addition, when the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the above-mentioned process performed by the change unit 4023 to change the environment in which the image of the object (i.e., target object) was taken to a different environment may be performed based on a trained model whose coefficients are determined by a supervised learning method.

[0025] For example, the change unit 4023 uses a trained model (for example, a convolutional neural network) whose parameters are determined using training data, which is one type of machine learning, to predict the processing content when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401. Here, the trained models used by the change unit 4023 for each prediction will be described.

[0026] (Pre-trained model) The trained model will be described. The change unit 4023 predicts processing content based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object photographed by the imaging device 202). Here, the trained model will be described in which the change unit 4023 predicts processing content based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object photographed by the imaging device 202).

[0027] In this case, image data of an object photographed by the photographing device 202 becomes one of the inputs. Furthermore, processing details actually set for the image data become one of the output data. A combination of input data and output data corresponding to that input data becomes one of the training data. For example, before the change unit 4023 predicts processing details, output data (i.e., processing details actually set for the image data of the object photographed by the photographing device 202) is identified for input data used by another device to predict processing details. Alternatively, output data is identified for input data by, for example, conducting experiments or simulations. In this way, training data consisting of a plurality of data combining input data and output data can be prepared. Note that training data is data used to determine parameter values ​​in a learning model in which parameter values ​​have not been determined.

[0028] 5 is a diagram showing an example of training data in the first embodiment of the present disclosure. Input data, which is image data of an object, and output data (i.e., processing content) corresponding to the input data form one set of data. In the example shown in FIG. 5, the training data includes 10,000 sets of data.

[0029] For example, consider the case where parameters in a learning model are determined using training data consisting of 10,000 sets of data as shown in FIG. 5. In this case, the training data is divided into, for example, training data, validation data, and test data. Examples of the ratios of training data, validation data, and test data include 70%, 15%, 15%, 95%, 2.5%, and 2.5%. For example, suppose that the training data consisting of data #1 to #10,000 is divided into training data (data #1 to #7000), validation data (data #7001 to #8500), and 15% test data (data #8501 to #10,000). In this case, data #1, which is training data, is input to a convolutional neural network, which is a learning model. The convolutional neural network outputs the processing content actually set for the image data of the object. Input data of the training data is input to the convolutional neural network, and each time the processing content actually set for the image data of the object is output from the convolutional neural network (in this case, each time each data of data #1 to #7000 is input to the convolutional neural network), parameters indicating the weighting of the data connections between nodes are changed by, for example, backpropagation according to the output (i.e., the model of the convolutional neural network is changed).In this way, training data is input to the neural network and the parameters are adjusted.

[0030] Next, the input data of the evaluation data (data #7001 to #8500) is input in order to the convolutional neural network whose parameters have been changed by the training data. The convolutional neural network outputs the processing content actually set for the image data of the object according to the input evaluation data. Here, if the data output by the convolutional neural network differs from the output data associated with the input data in FIG. 5, the parameters are changed so that the output of the convolutional neural network becomes the output data associated with the input data in FIG. 5. The convolutional neural network (i.e., the learning model) whose parameters have been determined in this way is the trained model.

[0031] Next, as a final check, test data (data #8501 to #10000) are input sequentially to the convolutional neural network of the trained model. The convolutional neural network of the trained model outputs the processing content actually set for the image data of the object according to the input test data. If the output data output by the convolutional neural network of the trained model for all test data matches the output data associated with the input data in FIG. 5, the convolutional neural network of the trained model is the desired model. Furthermore, if the output data output by the convolutional neural network of the trained model for even one of the test data does not match the output data associated with the input data in FIG. 5, new training data is used to determine the parameters of the trained model. The above-described determination of the parameters of the trained model is repeated until a trained model having the desired parameters is obtained. When a trained model having the desired parameters is obtained, the trained model is recorded in the modification unit 4023. The modification unit 4023 may then predict the processing content using this trained model.

[0032] (Processing performed by the robot system) 6 is a diagram showing an example of a processing flow of the robot system 1 according to the first embodiment of the present disclosure. Next, processing performed by the robot system 1 will be described with reference to FIG.

[0033] The photographing device 202 photographs an object on the tray T (step S1). For example, the photographing device 202 photographs multiple types of objects (e.g., products) packaged in transparent packaging material, and barcodes or tags attached to the objects that indicate the identity of the objects.

[0034] The identification unit 4021 identifies the type of object based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object captured by the imaging device 202) (step S2). For example, the identification unit 4021 compares images of each of a plurality of types of objects prepared in advance with the image of the object acquired by the acquisition unit 401. Then, the identification unit 4021 identifies the object shown in the image acquired by the acquisition unit 401 as the object shown in the prepared image with the largest matching image portion. Also, for example, the identification unit 4021 reads a barcode or tag attached to the object shown in the image acquired by the acquisition unit 401. Then, if the identification unit 4021 can identify the object by reading it, it identifies the object shown in the image acquired by the acquisition unit 401 as that object.

[0035] If the identification unit 4021 can identify the type of object indicated by the image acquired by the acquisition unit 401 (YES in step S2), the control unit 4022 controls the drive mechanism 203 to grip the identified object (step S3).

[0036] If the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401 (NO in step S2), the change unit 4023 changes the environment in which the image was captured to a different environment (step S4). For example, the change unit 4023 predicts processing content using a trained model. The change unit 4023 changes the environment in which the image was captured to a different environment based on the predicted processing content. Then, the change unit 4023 returns to the processing of step S1.

[0037] (advantage) The robot system 1 according to the first embodiment of the present disclosure has been described above. The robot system 1 includes a robot arm 201 capable of grasping an object including a transparent packaging material and an object to be packaged in the packaging material. In the robot system 1, a drive mechanism 203 drives the robot arm 201. An identification unit 4021 identifies the type of object based on image processing of an image of the object. If the identification unit 4021 cannot identify the type of object, a change unit 4023 changes the environment in which the image of the object was captured to a different environment.

[0038] This robot system 1 can change the environment in which the camera 202 captures an image of an object. This change in the environment changes the image of the object captured by the camera 202. As a result, the image of the object captured by the camera 202 may be improved to the extent that the object can be identified.

[0039] <First Modification of First Embodiment> Next, a robot system 1 according to a first modified example of the first embodiment of the present disclosure will be described. FIG. 7 is a diagram illustrating an example of the configuration of a robot 20 according to the first modified example of the first embodiment. In the first embodiment, the robot 20 has been described as a single-arm robot having a robot arm 201. However, in the first modified example of the first embodiment, the robot 20 may have a robot arm 204 (an example of a second robot arm) in addition to the robot arm 201, as shown in FIG. 7. The number of robot arms 204 may be two or more. In this case, when the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the change unit 4023 may move the robot arm 204 as an object that changes the refractive index of light between the object (i.e., the target object) and the lighting device 30. In the robot system 1 including this robot 20, the robot arm 204 can be used as an object that changes the refractive index of light. As a result, there is a possibility that the object can be identified to an extent that it can be identified.

[0040] <Second Modification of First Embodiment> Next, a robot system 1 according to a second modification of the first embodiment of the present disclosure will be described. In the second modification of the first embodiment of the present disclosure, the change unit 4023 of the processing device 40 may store a correspondence between the processing content and the image performed after the environment is changed. The change unit 4023 of the processing device 40 may perform additional learning to change the parameters of the trained model using the stored correspondence between the processing content and the image as input data. In other words, the trained model may be changed based on the changed environment. Note that this additional learning may be performed in real time when the processing content is performed. Alternatively, this additional learning may be performed after a certain amount of data has been collected. Alternatively, this additional learning may use data from a robot system 1 located in a different location. A robot system 1 including this robot 20 can use a trained model whose parameters are determined based on the latest processing data. As a result, improved accuracy in identifying an object when an image of the object captured by the imaging device 202 is used.

[0041] Second Embodiment Next, a robot system 1 according to a second embodiment of the present disclosure will be described. Similar to the robot system 1 according to the first embodiment of the present disclosure shown in FIG. 1, the robot system 1 includes a transport device 10, a robot 20, a lighting device 30, and a processing device 40. Similar to the processing device 40 according to the first embodiment of the present disclosure shown in FIG. 3, the processing device 40 includes an acquisition unit 401 and a processing unit 402. Similar to the processing device 402 according to the first embodiment of the present disclosure shown in FIG. 4, the processing unit 402 includes an identification unit 4021 (an example of an identification unit), a control unit 4022, and a change unit 4023 (an example of a change unit). However, in the first embodiment of the present disclosure, the change unit 4023 was described as predicting processing content using a trained model. However, in the second embodiment of the present disclosure, the change unit 4023 attempts to change the processing content from the environment in which the image was captured to a different environment by trial and error. That is, if the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401, the change unit 4023 performs a first processing content, and if the identification unit 4021 cannot identify the type of object from an image of the target object captured after the first processing content is performed, the change unit 4023 performs a second processing content different from the first processing content. For example, if there are five processing contents, namely, processing A, processing B, processing C, processing D, and processing E, the change unit 4023 may try to change the environment in which the image was captured to a different environment by trying the processing contents in a predetermined order (for example, in the order of processing A, processing B, processing C, processing D, and processing E) or in a random order in the processing of step S4 of the processing flow shown in FIG. 6.

[0042] (advantage) The robot system 1 according to the second embodiment of the present disclosure has been described above. In the robot system 1, the change unit 4023 performs a first processing content when the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, and performs a second processing content different from the first processing content when the identification unit 4021 cannot identify the type of object based on an image of the target object captured after the first processing content is performed.

[0043] With this robot system 1, when the number of types of processing content is small, it is possible to easily change the environment in which the image capturing device 202 captures an object without having to prepare a trained model in advance. This change in the environment changes the image of the object captured by the image capturing device 202. As a result, there is a possibility that the image of the object captured by the image capturing device 202 may be improved to the extent that the object can be identified.

[0044] <Modification of the second embodiment> Next, a robot system 1 according to a modification of the second embodiment of the present disclosure will be described. The robot system 1 according to the modification of the second embodiment includes a transport device 10, a robot 20, a lighting device 30, and a processing device 40, similar to the robot system 1 according to the second embodiment. The processing device 40 includes an acquisition unit 401 and a processing unit 402, similar to the processing device 40 according to the second embodiment of the present disclosure. The processing unit 402 includes an identification unit 4021 (an example of an identification unit), a control unit 4022, and a change unit 4023 (an example of a change unit), similar to the processing unit 402 according to the second embodiment of the present disclosure. However, in the second embodiment of the present disclosure, the change unit 4023 has been described as attempting to change the processing content by trial and error. However, in the modification of the second embodiment of the present disclosure, if the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 may predict the processing content based on the image of the object captured by the imaging device 202 using an image processing method other than the method using a trained model. FIG. 8 is a diagram illustrating an example of a data table TBL according to a modified example of the second embodiment of the present disclosure. For example, a data table TBL as shown in FIG. 8 is prepared, in which an image acquired by the acquisition unit 401 is associated with a processing content when the identification unit 4021 is unable to identify the type of object indicated by the image. The change unit 4023 compares the image acquired by the acquisition unit 401 when the identification unit 4021 is unable to identify the type of object indicated by the image with each image in the data table TBL. The change unit 4023 then identifies an image in the data table TBL that is closest to the image acquired by the acquisition unit 401 when the identification unit 4021 is unable to identify the type of object indicated by the image. For example, the change unit 4023 compares parts of the images and identifies the image with the largest number of matches as the closest image. The change unit 4023 then identifies a processing content associated with the identified image in the data table TBL. The change unit 4023 then executes the identified processing content.

[0045] (advantage) The robot system 1 according to the modified example of the second embodiment of the present disclosure has been described above. In the robot system 1, if the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the change unit 4023 predicts processing content based on the image of the object captured by the imaging device 202 using an image processing method different from the method using a trained model.

[0046] In another embodiment of the present disclosure, the image capturing device 202 does not have to be provided on the robot arm 201. For example, the image capturing device 202 may be provided above the tray T.

[0047] With this robot system 1, the change unit 4023 can identify a processing content that is more likely than when processing content is tried by trial and error. Based on the identified processing content, the change unit 4023 changes the environment in which the image capturing device 202 captures an image of an object. This change in the environment changes the image of the object captured by the image capturing device 202. As a result, there is a possibility that the image of the object captured by the image capturing device 202 will be improved to an extent that the object can be identified.

[0048] A robot system 1 with a minimum configuration according to an embodiment of the present disclosure will be described. FIG. 9 is a diagram illustrating the robot system 1 with a minimum configuration according to an embodiment of the present disclosure. The robot system 1 with a minimum configuration according to an embodiment of the present disclosure includes a robot arm 201, a driving mechanism 203 (an example of a driving means), an identifying unit 4021 (an example of an identifying means), and a changing unit 4023 (an example of an changing means). The robot arm 201 is capable of grasping an object including a transparent packaging material and an object packaged in the packaging material. The robot arm 201 can be realized, for example, using the functions of the robot arm 201 illustrated in FIG. 1. The driving mechanism 203 drives the robot arm 201. The driving mechanism 203 can be realized, for example, using the functions of the driving mechanism 203 illustrated in FIG. 1. The identifying unit 4021 identifies the type of object based on image processing of an image of the object. The identifying unit 4021 can be realized, for example, using the functions of the identifying unit 4021 illustrated in FIG. 4. If the identification unit 4021 cannot identify the type of object, the change unit 4023 changes the environment in which the image of the target object was captured to a different environment. The change unit 4023 can be realized, for example, by using the function of the change unit 4023 illustrated in FIG.

[0049] Next, a description will be given of the processing of the robot system 1 with the minimum configuration. Fig. 10 is a diagram showing an example of a processing flow of the robot system 1 with the minimum configuration. Here, the processing of the robot system 1 with the minimum configuration will be described with reference to Fig. 10.

[0050] The robot arm 201 can grasp an object including a transparent packaging material and an object packaged in the packaging material. The drive mechanism 203 drives the robot arm 201 (step S101). The identification unit 4021 identifies the type of object based on image processing of the image of the object (step S102). If the identification unit 4021 cannot identify the type of object, the change unit 4023 changes the environment in which the image of the object was captured to a different environment (step S103). In this way, the robot system 1 can change the environment in which an image of an object packaged in a transparent packaging material is captured to an environment in which an image that identifies the object can be captured. As a result, the robot system 1 can appropriately and accurately recognize the object even if the object is packaged in a transparent packaging material.

[0051] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0052] Although the embodiments of the present disclosure have been described, the robot system 1, the robot 20, the processing device 40, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0053] 11 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 11, the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the robot system 1, the robot 20, the processing device 40, and other control devices described above are each implemented in the computer 5. The operations of the above-described processing units are stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to the above-described storage units in accordance with the program.

[0054] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. In addition, when this program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-mentioned processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0055] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0056] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0057] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0058] (Appendix 1) a robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; A robot system comprising:

[0059] (Appendix 2) an imaging means capable of capturing an image of the object; Equipped with The change means is If the identification means cannot identify the type of the object, the imaging means is controlled so that the angle is different from the angle of the imaging means that captured the image of the target object. 10. The robotic system of claim 1.

[0060] (Appendix 3) illumination means for illuminating the object; Equipped with The change means is If the identification means cannot identify the type of the object, the state of light irradiated on the object when the image of the object was captured is changed to a state different from the state of light irradiated on the object when the image of the object was captured. 10. The robotic system of claim 1.

[0061] (Appendix 4) The change means is If the identification means cannot identify the type of the object, the illumination means is controlled so that the angle of the light is different from the angle of the light in a state where the image of the object was captured. 4. The robotic system of claim 3.

[0062] (Appendix 5) The change means is If the identification means cannot identify the type of the object, an object that changes the refractive index of light is moved between the target object and the illumination means. 4. The robotic system of claim 3.

[0063] (Appendix 6) a second robotic arm separate from the robotic arm; Equipped with The change means is If the identification means cannot identify the type of the object, the second robot arm is moved between the target object and the lighting means. 6. The robotic system of claim 5.

[0064] (Appendix 7) The change means is If the identification means cannot identify the type of the object, the drive means is controlled so that the state of the object changes. 10. The robotic system of claim 1.

[0065] (Appendix 8) The change means is If the identification means cannot identify the type of the object, the drive means is controlled so as to change the orientation of the object. 8. The robotic system of claim 7.

[0066] (Appendix 9) The change means is If the identification means cannot identify the type of the object, the drive means is controlled so that the state of the packaging member changes. 8. The robotic system of claim 7.

[0067] (Appendix 10) The change means is If the identification means cannot identify the type of the object, the drive means is controlled to suppress the expansion of the packaging member. 10. The robotic system of claim 9.

[0068] (Appendix 11) The change means is If the identification means cannot identify the type of the object, the drive means is controlled to stretch the packaging member. 10. The robotic system of claim 9.

[0069] (Appendix 12) The change means is If the identification means cannot identify the type of the object, the environment in which the image of the object was captured is changed to a different environment based on a trained model whose coefficients are determined by a supervised learning method. 12. The robot system according to any one of claims 1 to 11.

[0070] (Appendix 13) The change means is Changing the trained model based on the changed environment, and changing the environment from the environment in which the image of the object was captured to a different environment based on the changed trained model. 13. The robotic system of claim 12.

[0071] (Appendix 14) The change means is If the identification means cannot identify the type of the object, a first processing content is carried out, and if the identification means cannot identify the type of the object based on an image of the object captured after the first processing content is carried out, a second processing content different from the first processing content is carried out. 12. The robot system according to any one of claims 1 to 11.

[0072] (Appendix 15) A processing method performed by a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, the method comprising: Drive the robot arm; identifying the type of object based on image processing of the image of the object; If the type of the object cannot be identified, the environment in which the image of the object was taken is changed to a different environment. Processing method.

[0073] (Appendix 16) A computer included in a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; A recording medium on which a program for executing the above is stored.

[0074] (Appendix 17) A robotic arm, a control means for controlling the operation of the robot arm so that the robot arm performs an operation on the object based on a recognition result of an image of the object photographed by an imaging device; Equipped with the object is an object packaged in a transparent packaging material, When the control means cannot recognize the object from the image, the control means controls the robot arm to change the environment in which the photographing device photographs the target object. Robot system. [Industrial Applicability]

[0075] According to each aspect of the present disclosure, even if an object (for example, a product) is packaged in a packaging material that has transparency, the object can be appropriately and accurately recognized. [Explanation of symbols]

[0076] 1. Robot System 5. Computer 6 CPU 7. Main memory 8. Storage 9. Interface 10. Conveying device 20. Robot 30. Lighting equipment 40 Processing equipment 101...Transport mechanism 201 Robot Arm 202 Imaging device 203 Drive mechanism 401...Acquisition Department 402 Processing section DB...Database T···Tray TBL...Data Table

Claims

1. a robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; an imaging means capable of capturing an image of the object; Equipped with The change means is If the identification means cannot identify the type of the object, the imaging means is controlled so that the angle is different from the angle of the imaging means that captured the image of the target object. Robot system.

2. A robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; an illumination means for illuminating the object; Equipped with The change means is If the identification means cannot identify the type of the object, the illumination means is controlled to change the state of light irradiated on the object so that it is different from the state of light irradiated on the object when the image of the object was captured, and to change the angle of the light so that it is different from the angle of the light when the image of the object was captured. Robot system.

3. A robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; an illumination means for illuminating the object; Equipped with The change means is If the identification means cannot identify the type of the object, the state of light irradiated on the object when the image of the object is captured is changed to a state different from that of the light irradiated on the object when the image of the object is captured, and an object that changes the refractive index of light is moved between the object and the illumination means. Robot system.

4. A robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; Equipped with The change means is If the identification means cannot identify the type of the object, the drive means is controlled so as to change the orientation of the object. Robot system.

5. A robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member; a driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of the image of the object; a change means for changing an environment in which an image of the object was taken to a different environment when the identification means is unable to identify the type of the object; Equipped with The change means is If the identification means cannot identify the type of the object, a first processing content is carried out, and if the identification means cannot identify the type of the object based on an image of the target object taken after the first processing content is carried out, a second processing content different from the first processing content is carried out. Robot system.

6. A processing method performed by a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, the method comprising: Drive the robot arm; identifying the type of object based on image processing of the image of the object; If the type of the object cannot be identified, the environment in which the image of the object was taken is changed to a different environment; If the type of the object cannot be identified, the photographing means is controlled so that the photographing angle is different from the angle of the photographing means that photographed the image of the target object. Processing method.

7. A computer included in a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; If the type of the object cannot be identified, controlling the photographing means so that the photographing angle is different from the angle of the photographing means that photographed the image of the target object; A program that executes the following.

8. A computer of a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; If the type of the object cannot be identified, change the state of light irradiated on the object so that it is different from the state of light irradiated on the object when the image of the object was taken, and control the lighting means that illuminates the object so that the angle of the light is different from the angle of the light when the image of the object was taken; A program that executes the following.

9. A computer having a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; If the type of the object cannot be identified, change the state of light irradiated on the object so that it is different from the state of light irradiated on the object when the image of the object is captured, and move an object that changes the refractive index of light between the object and the lighting means that illuminates the object; A program that executes the following.

10. A robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; If the type of the object cannot be identified, controlling a driving means that drives the robot arm so as to change the orientation of the target object; A program that executes the following.

11. A robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the object type based on image processing of the image of the object; If the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; If the type of the object cannot be identified, a first processing content is performed, and if the type of the object cannot be identified from an image of the target object after the first processing content is performed, a second processing content different from the first processing content is performed; A program that executes the following.

Images