ROBOT SYSTEM, PROCESSING METHOD, AND PROGRAM

JPWO2023188045A5Active Publication Date: 2024-12-04NEC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024510836
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-09-26
Publication Date
2024-12-04
Estimated Expiration
2042-03-29

AI Technical Summary

Technical Problem

Robots struggle to accurately recognize objects packaged in transparent packaging due to light reflection and refractive issues, making it difficult to perform appropriate actions.

Method used

A robot system equipped with a robot arm, identification means through image processing, and a change mechanism that adjusts the environment, such as altering the object's orientation or lighting, to improve image recognition.

Benefits of technology

Enables accurate recognition and appropriate action on objects packaged in transparent packaging by modifying the environment to reduce reflection and refractive interference, enhancing image clarity.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

This robot system comprises: a robot arm that is capable of grasping an object, the object including a packaging member having transparency and a body of matter to be packaged with the packaging member; a drive means that drives the robot arm; an identification means that identifies a type of the body of matter on the basis of image processing on an image of the object; and a change means that changes, if the identification means fails to identify the type of the substance, changes an environment in which the image of the object was taken to a different environment.
Need to check novelty before this filing date? Find Prior Art

Description

ROBOT SYSTEM, PROCESSING METHOD, AND RECORDING MEDIUM

[0001] The present disclosure relates to a robot system, a processing method, and a recording medium.

[0002] Robots are used in various fields such as logistics. Japanese Patent Application Laid-Open Nos. 2003-222299 and 2003-222299 disclose related technologies relating to a robot system that grasps an object.

[0003] JP 2018-176334 A Patent No. 6752615 A

[0004] However, in an image of an object such as a product covered with a transparent packaging material, there is a possibility that light is reflected by the packaging material or surrounding objects are reflected in the image. Therefore, it may be difficult to identify the object from such an image. Even if the robots disclosed in Patent Documents 1 and 2 are used, they cannot accurately recognize the object in such a situation, and therefore cannot take appropriate action on the object.

[0005] One of the objectives of each aspect of the present disclosure is to provide a robot system, a processing method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, a robot system includes a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material, a driving means for driving the robot arm, an identification means for identifying the type of the object based on image processing of an image of the object, and a change means for changing the environment in which the image of the object was captured to a different environment if the identification means is unable to identify the type of the object.

[0007] According to another aspect of the present disclosure, a processing method is a processing method performed by a robot system including a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material, the processing method driving the robot arm to identify a type of the object based on image processing of an image of the object, and if the type of the object cannot be identified, changing an environment from the environment in which the image of the object was taken to a different environment.

[0008] According to another aspect of the present disclosure, the recording medium stores a program that causes a computer of a robot system having a robot arm capable of grasping an object including a transparent packaging material and an object packaged by the packaging material to drive the robot arm, identify the type of the object based on image processing of an image of the object, and, if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment.

[0009] According to another aspect of the present disclosure, a robot system includes a robot arm and a control means for controlling the operation of the robot arm so that the robot arm performs an action on an object based on a recognition result of an image of the object photographed by an imaging device, wherein the object is an object packaged in a transparent packaging material, and when the control means cannot recognize the object from the image, the control means controls the robot arm to change the environment in which the imaging device photographs the object.

[0010] According to each aspect of the present disclosure, even if an object is packaged in a packaging material that has transparency, the object can be appropriately and accurately recognized.

[0011] FIG. 1 is a diagram illustrating an example of a configuration of a robot system according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a database according to the first embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a configuration of a processing device according to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of a configuration of a processing device according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of teacher data in the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of a processing flow of a robot system according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a configuration of a robot according to a first modified example of the first embodiment. FIG. 8 is a diagram illustrating an example of a data table in a modified example of the second embodiment of the present disclosure. FIG. 9 is a diagram illustrating a robot system with a minimum configuration according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a processing flow of a robot system with a minimum configuration. FIG. 11 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

[0012] The following describes the embodiments in detail with reference to the drawings. A robot system 1 according to each embodiment of the present disclosure can change the environment in which an image of an object (e.g., a product) packaged in a transparent packaging material is photographed to an environment in which an image that identifies the object can be captured. This change in environment includes, in the robot system 1 described below, actions of the robot 20 relative to the packaging material, such as stretching the packaging material, and actions of the robot 20 that change the state of the object, such as changing the orientation of the object. <First Embodiment> A robot system 1 according to a first embodiment of the present disclosure can appropriately and accurately recognize an object (e.g., a product) even if the object is packaged in a transparent packaging material. Examples of transparent packaging materials include plastic wrap, vinyl, and plastic containers. The robot system 1 identifies an object to be grasped from among multiple types of objects based on image processing of the captured image. Note that, in the present disclosure, grasping includes not only holding an object at the position of the robot arm by clamping it, but also holding the object at the position of the robot arm by suction. The robot system 1 may identify an object to be subjected to an action from among multiple types of objects based on image processing of the captured image. The operation does not have to be the gripping described above, but may be, for example, rotation, movement, opening, boxing, etc. The operation is not limited to the examples described above. The robot system 1 is used, for example, in warehouses, food factories, supermarkets, convenience stores, etc.

[0013] 1 is a diagram illustrating an example of the configuration of a robot system 1 according to a first embodiment of the present disclosure. As shown in FIG. 1, the robot system 1 includes a transport device 10, a robot 20, an illumination device 30 (an example of an illumination means), and a processing device 40.

[0014] As shown in FIG. 1 , the conveying device 10 includes a conveying mechanism 101, a tray T, and a database DB. Based on the database DB, the conveying mechanism 101 places multiple types of objects (e.g., products) packaged in transparent packaging on the tray T and moves the tray T to a position where the robot 20 can grasp the objects. For example, products (e.g., product A, product B, and product C) received at a warehouse, food factory, supermarket, convenience store, etc. are placed on the tray T, and information indicating the type and quantity of the received products is recorded for each tray T in the database DB. FIG. 2 is a diagram illustrating an example of the database DB according to the first embodiment of the present disclosure. In the example of the database DB illustrated in FIG. 2 , three types of products, namely product A, product B, and product C, are placed on the tray T1, and it is shown that there is one quantity of product A, two quantities of product B, and three quantities of product C. When the robot 20 is to grasp product A as an object, the conveying mechanism 101 identifies the tray T1 associated with product A in the database DB. Then, the transport mechanism 101 moves the tray T1 to a position where the robot 20 can grasp it.

[0015] As shown in FIG. 1 , the robot 20 includes a robot arm 201, a camera 202 (an example of a camera means), and a drive mechanism 203 (an example of a drive means). The robot arm 201 grasps an object in accordance with the operation of the drive mechanism 203. The camera 202 takes an image of the object on the tray T. For example, the camera 202 takes an image of multiple types of objects (e.g., products) packaged in a transparent packaging material, or a barcode or tag attached to the object that indicates the identity of the object. Examples of the camera 202 include a camera and a video camera. The drive mechanism 203 operates the robot arm 201 under the control of the processing device 40. The lighting device 30 illuminates the object placed on the tray T.

[0016] 3 is a diagram illustrating an example of the configuration of the processing device 40 according to the first embodiment of the present disclosure. As shown in FIG. 3, the processing device 40 includes an acquisition unit 401 and a processing unit 402. The acquisition unit 401 acquires an image of an object captured by the imaging device 202.

[0017] 4 is a diagram illustrating an example of the configuration of the processing unit 402 according to the first embodiment of the present disclosure. As shown in FIG. 4, the processing unit 402 includes an identifying unit 4021 (an example of an identifying means), a control unit 4022 (an example of a control means), and a changing unit 4023 (an example of a changing means, an example of a control means).

[0018] The identification unit 4021 identifies the type of object based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object captured by the imaging device 202). For example, the identification unit 4021 compares the image of the object acquired by the acquisition unit 401 with images of each of multiple types of objects prepared in advance. The identification unit 4021 identifies the type of object in the image acquired by the acquisition unit 401 based on the comparison result. Alternatively, the identification unit 4021 determines that the type of object cannot be identified based on the comparison result. For example, the identification unit 4021 may identify the type of object in the image acquired by the acquisition unit 401 by applying a model created by machine learning, such as a neural network, to the image acquired by the acquisition unit 401. Furthermore, for example, the identification unit 4021 may identify the object shown in the image acquired by the acquisition unit 401 as the object shown in the prepared image with the most matching image portions. Furthermore, for example, the identification unit 4021 reads a barcode or tag attached to the object shown in the image acquired by the acquisition unit 401. Then, when the identification unit 4021 can identify the object by reading it, it may identify that object as the object shown in the image acquired by the acquisition unit 401. In other words, the above-described processing can also be said to be processing in which the identification unit 4021 analyzes an image and executes an operation to recognize the object shown in the image.

[0019] The control unit 4022 controls the drive mechanism 203 to grip the identified object when the identification unit 4021 can identify the type of object indicated by the image acquired by the acquisition unit 401. In other words, the above-described processing can also be said to be processing in which, when an object is recognized from an image, the control unit 4022 controls the operation of the robot arm 201 based on the recognition result so that the robot arm 201 performs an operation (for example, a gripping operation) on the target object.

[0020] The change unit 4023 changes the environment from that in which the image was taken to a different environment when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401. Examples of reasons why the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401 include the refractive index of the transparent packaging material, reflections (light source, glare) on the surface of the transparent packaging material, and the position and orientation of the object within the transparent packaging material.

[0021] The following is an example of a process for resolving the inability of the identification unit 4021 to identify the type of object indicated by the image acquired by the acquisition unit 401. For example, when the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the change unit 4023 controls the image capturing device 202 so that the angle is different from the angle of the image capturing device 202 that captured the image of the object (i.e., the target object).

[0022] Furthermore, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 changes the state of light irradiated on the object (i.e., target object) in the state in which the image was captured so that it is different from the state of light irradiated on the object (i.e., the target object). Specifically, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 changes the angle of light irradiated on the object (i.e., the target object) in the state in which the image was captured so that it is different from the angle of light irradiated on the object (i.e., the target object). Specifically, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 moves an object that changes the refractive index of light between the object (i.e., the target object) and the lighting device 30. Specifically, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the state of the object (i.e., the target object). More specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the orientation of the object (i.e., the target object). Also, more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to change the state of the packaging material. Even more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to suppress the bulge of the packaging material. Also, even more specifically, for example, when the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 controls the drive mechanism 203 to stretch the packaging material.

[0023] Furthermore, the processing of the change unit 4023 may be realized by the control unit 4022 controlling the operation of the robot arm 201. When the target object is an object wrapped in a transparent wrapping material, the above-described processing can also be said to be processing in which the control unit 4022 controls the operation of the robot arm 201 so as to change the environment in which the image capturing device captures the target object if the object cannot be recognized from the image.

[0024] In addition, when the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the above-mentioned process performed by the change unit 4023 to change the environment in which the image of the object (i.e., target object) was taken to a different environment may be performed based on a learned model whose coefficients are determined by a supervised learning technique.

[0025] For example, the change unit 4023 uses a trained model (e.g., a convolutional neural network) whose parameters are determined using teacher data, which is one type of machine learning, to predict the processing content when the identification unit 4021 cannot identify the type of object shown in the image acquired by the acquisition unit 401. Here, the trained models used by the change unit 4023 for each prediction will be described.

[0026] (Trained Model) The trained model will be described. The change unit 4023 predicts processing content based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object photographed by the imaging device 202). Here, a trained model will be described in which the change unit 4023 predicts processing content based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object photographed by the imaging device 202).

[0027] In this case, image data of an object photographed by the photographing device 202 becomes one of the inputs. Furthermore, the processing content actually set for the image data becomes one of the output data. A combination of input data and output data corresponding to the input data becomes one of the training data. For example, before the change unit 4023 predicts the processing content, output data (i.e., processing content actually set for the image data of the object photographed by the photographing device 202) is identified for the input data used by another device to predict the processing content. Alternatively, output data is identified for the input data by, for example, conducting experiments or simulations. In this way, training data consisting of multiple data combining input data and output data can be prepared. Training data is data used to determine parameter values ​​in a learning model in which parameter values ​​have not yet been determined.

[0028] 5 is a diagram illustrating an example of training data according to the first embodiment of the present disclosure. One set of data is input data, which is image data of an object, and output data (i.e., processing content) corresponding to the input data. In the example illustrated in FIG. 5, the training data includes 10,000 sets of data.

[0029] For example, consider the case where parameters in a learning model are determined using training data consisting of 10,000 sets of data as shown in FIG. 5. In this case, the training data is divided into, for example, training data, validation data, and test data. Examples of the ratios of training data, validation data, and test data include 70%, 15%, 15%, 95%, 2.5%, and 2.5%. For example, suppose that the training data of data #1 to #10,000 is divided into training data (data #1 to #7000), validation data (data #7001 to #8500), and 15% test data (data #8501 to #10,000). In this case, data #1, which is training data, is input into a convolutional neural network, which is a learning model. The convolutional neural network outputs the processing content actually set for the image data of the object. Input data from the training data is input to the convolutional neural network, and each time the processing content actually set for the image data of the object is output from the convolutional neural network (in this case, each time each of data #1 to #7000 is input to the convolutional neural network), parameters indicating the weighting of the data connections between nodes are changed (i.e., the model of the convolutional neural network is changed) by performing backpropagation, for example, in response to the output. In this way, training data is input to the neural network and the parameters are adjusted.

[0030] Next, the input data of the evaluation data (data #7001 to #8500) are sequentially input to the convolutional neural network whose parameters have been changed by the training data. The convolutional neural network outputs the processing content actually set for the image data of the object according to the input evaluation data. Here, if the data output by the convolutional neural network differs from the output data associated with the input data in FIG. 5, the parameters are changed so that the output of the convolutional neural network becomes the output data associated with the input data in FIG. 5. The convolutional neural network (i.e., the learning model) whose parameters have been determined in this way is the trained model.

[0031] Next, as a final check, the input data of the test data (data #8501 to #10000) is input sequentially to the convolutional neural network of the trained model. The convolutional neural network of the trained model outputs the processing content actually set for the image data of the object according to the input test data. If the output data output by the convolutional neural network of the trained model for all test data matches the output data associated with the input data in FIG. 5, the convolutional neural network of the trained model is the desired model. Furthermore, if the output data output by the convolutional neural network of the trained model for even one of the test data does not match the output data associated with the input data in FIG. 5, new training data is used to determine the parameters of the trained model. The above-described determination of the parameters of the trained model is repeated until a trained model having the desired parameters is obtained. When a trained model having the desired parameters is obtained, the trained model is recorded in the modification unit 4023. The modification unit 4023 may then predict the processing content using this trained model.

[0032] (Processing Performed by Robot System) Fig. 6 is a diagram showing an example of a processing flow of the robot system 1 according to the first embodiment of the present disclosure. Next, processing performed by the robot system 1 will be described with reference to Fig. 6 .

[0033] The photographing device 202 photographs an object on the tray T (step S1). For example, the photographing device 202 photographs multiple types of objects (e.g., products) packaged in transparent packaging material, and barcodes or tags attached to the objects that indicate the identity of the objects.

[0034] The identification unit 4021 identifies the type of object based on the image of the object acquired by the acquisition unit 401 (i.e., the image of the object captured by the imaging device 202) (step S2). For example, the identification unit 4021 compares images of each of a plurality of types of objects prepared in advance with the image of the object acquired by the acquisition unit 401. Then, the identification unit 4021 identifies the object shown in the image acquired by the acquisition unit 401 as the object shown in the prepared image with the largest number of matching image portions. Also, for example, the identification unit 4021 reads a barcode or tag attached to the object shown in the image acquired by the acquisition unit 401. Then, if the identification unit 4021 can identify the object by reading it, it identifies the object shown in the image acquired by the acquisition unit 401 as that object.

[0035] If the identification unit 4021 can identify the type of object indicated by the image acquired by the acquisition unit 401 (YES in step S2), the control unit 4022 controls the drive mechanism 203 to grasp the identified object (step S3).

[0036] If the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401 (NO in step S2), the change unit 4023 changes the environment in which the image was captured to a different environment (step S4). For example, the change unit 4023 predicts processing content using a trained model. The change unit 4023 changes the environment in which the image was captured to a different environment based on the predicted processing content. Then, the change unit 4023 returns to the processing of step S1.

[0037] (Advantages) The robot system 1 according to the first embodiment of the present disclosure has been described above. The robot system 1 includes a robot arm 201 capable of grasping an object including a transparent packaging material and an object to be packaged in the packaging material. In the robot system 1, a drive mechanism 203 drives the robot arm 201. An identification unit 4021 identifies the type of object based on image processing of an image of the object. If the identification unit 4021 cannot identify the type of object, a change unit 4023 changes the environment in which the image of the object was captured to a different environment.

[0038] The robot system 1 can change the environment in which the image capturing device 202 captures an image of an object. This change in the environment changes the image of the object captured by the image capturing device 202. As a result, the image of the object captured by the image capturing device 202 may be improved to the extent that the object can be identified.

[0039] <First Modification of First Embodiment> Next, a robot system 1 according to a first modification of the first embodiment of the present disclosure will be described. FIG. 7 is a diagram illustrating an example of the configuration of a robot 20 according to the first modification of the first embodiment. In the first embodiment, the robot 20 has been described as a single-arm robot having a robot arm 201. However, in the first modification of the first embodiment, the robot 20 may have a robot arm 204 (an example of a second robot arm) in addition to the robot arm 201, as shown in FIG. 7 . Furthermore, the robot arm 204 may include two or more robot arms. In this case, if the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the change unit 4023 may move the robot arm 204 between the object (i.e., the target object) and the lighting device 30 as an object that changes the refractive index of light. In the robot system 1 including this robot 20, the robot arm 204 can be used as an object that changes the refractive index of light. As a result, the object may be identified to an extent that it can be identified.

[0040] <Second Modification of First Embodiment> Next, a robot system 1 according to a second modification of the first embodiment of the present disclosure will be described. In the second modification of the first embodiment of the present disclosure, the change unit 4023 of the processing device 40 may store a correspondence between the processing content and the image performed after the environment is changed. The change unit 4023 of the processing device 40 may perform additional learning to change the parameters of the trained model using the stored correspondence between the processing content and the image as input data. In other words, the trained model may be changed based on the changed environment. Note that this additional learning may be performed in real time when the processing content is performed. Alternatively, this additional learning may be performed after a certain amount of data has been collected. Alternatively, this additional learning may use data from a robot system 1 located in a different location. A robot system 1 including this robot 20 can use a trained model whose parameters are determined based on the latest processing data. As a result, improved accuracy in identifying an object when an image of the object captured by the imaging device 202 is captured can be expected.

[0041] Second Embodiment Next, a robot system 1 according to a second embodiment of the present disclosure will be described. Similar to the robot system 1 according to the first embodiment of the present disclosure shown in FIG. 1 , the robot system 1 includes a transport device 10, a robot 20, a lighting device 30, and a processing device 40. Similar to the processing device 40 according to the first embodiment of the present disclosure shown in FIG. 3 , the processing device 40 includes an acquisition unit 401 and a processing unit 402. Similar to the processing device 40 according to the first embodiment of the present disclosure shown in FIG. 4 , the processing unit 402 includes an identification unit 4021 (an example of an identification unit), a control unit 4022, and a change unit 4023 (an example of a change unit). However, in the first embodiment of the present disclosure, the change unit 4023 was described as predicting processing content using a trained model. However, in the second embodiment of the present disclosure, the change unit 4023 attempts to change the processing content from the environment in which the image was captured to a different environment by trial and error. That is, if the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, the change unit 4023 performs a first processing content, and if the identification unit 4021 cannot identify the type of object from an image of the target object captured after the first processing content is performed, the change unit 4023 performs a second processing content different from the first processing content. For example, if there are five processing contents, namely, processing A, processing B, processing C, processing D, and processing E, the change unit 4023 may attempt to change the environment in which the image was captured to a different environment by trying the processing contents in a predetermined order (for example, the order of processing A, processing B, processing C, processing D, and processing E) or in a random order in the processing of step S4 of the processing flow shown in FIG. 6 .

[0042] (Advantages) The robot system 1 according to the second embodiment of the present disclosure has been described above. In the robot system 1, the change unit 4023 performs a first process if the identification unit 4021 cannot identify the type of object indicated by the image acquired by the acquisition unit 401, and performs a second process different from the first process if the identification unit 4021 cannot identify the type of object based on an image of the target object captured after the first process is performed.

[0043] With this robot system 1, when the number of types of processing content is small, it is possible to easily change the environment in which the image capturing device 202 captures an object without having to prepare a trained model in advance. This change in the environment changes the image of the object captured by the image capturing device 202. As a result, the image of the object captured by the image capturing device 202 may be improved to the extent that the object can be identified.

[0044] <Modification of Second Embodiment> Next, a robot system 1 according to a modification of the second embodiment of the present disclosure will be described. The robot system 1 according to the modification of the second embodiment includes a transport device 10, a robot 20, a lighting device 30, and a processing device 40, similar to the robot system 1 according to the second embodiment. The processing device 40 includes an acquisition unit 401 and a processing unit 402, similar to the processing device 40 according to the second embodiment of the present disclosure. The processing unit 402 includes an identification unit 4021 (an example of an identification unit), a control unit 4022, and a change unit 4023 (an example of a change unit), similar to the processing unit 402 according to the second embodiment of the present disclosure. However, in the second embodiment of the present disclosure, the change unit 4023 has been described as attempting to change the processing content through trial and error. However, in the modification of the second embodiment of the present disclosure, if the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 may predict the processing content based on the image of the object captured by the imaging device 202 using an image processing method other than the method using a trained model. FIG. 8 is a diagram illustrating an example of the data table TBL in a modified example of the second embodiment of the present disclosure. For example, a data table TBL as shown in FIG. 8 is prepared, which associates an image acquired by the acquisition unit 401 with a processing content when the identification unit 4021 is unable to identify the type of object indicated by the image. The change unit 4023 compares the image acquired by the acquisition unit 401 when the identification unit 4021 is unable to identify the type of object indicated by the image with each image in the data table TBL. The change unit 4023 then identifies an image in the data table TBL that is closest to the image acquired by the acquisition unit 401 when the identification unit 4021 is unable to identify the type of object indicated by the image. For example, the change unit 4023 compares parts of the images and identifies the image with the largest number of matches as the closest image. The change unit 4023 then identifies a processing content associated with the identified image in the data table TBL. The change unit 4023 then executes the identified processing content.

[0045] (Advantages) The robot system 1 according to the modified example of the second embodiment of the present disclosure has been described above. In the robot system 1, if the identification unit 4021 cannot identify the type of object indicated in the image acquired by the acquisition unit 401, the change unit 4023 predicts processing content based on the image of the object captured by the imaging device 202 using an image processing method different from the method using a trained model.

[0046] In another embodiment of the present disclosure, the image capturing device 202 does not have to be provided on the robot arm 201. For example, the image capturing device 202 may be provided above the tray T.

[0047] With this robot system 1, the change unit 4023 can identify a processing content that is more likely than when processing content is attempted by trial and error. The change unit 4023 changes the environment in which the image capturing device 202 captures an image of an object based on the identified processing content. This change in the environment changes the image of the object captured by the image capturing device 202. As a result, there is a possibility that the image of the object captured by the image capturing device 202 will be improved to an extent that the object can be identified.

[0048] A robot system 1 with a minimum configuration according to an embodiment of the present disclosure will be described. FIG. 9 is a diagram illustrating the robot system 1 with a minimum configuration according to an embodiment of the present disclosure. The robot system 1 with a minimum configuration according to an embodiment of the present disclosure includes a robot arm 201, a drive mechanism 203 (an example of a drive means), an identification unit 4021 (an example of an identification means), and a change unit 4023 (an example of an change means). The robot arm 201 is capable of grasping an object including a transparent packaging material and an object packaged in the packaging material. The robot arm 201 can be realized, for example, using the functions of the robot arm 201 illustrated in FIG. 1. The drive mechanism 203 drives the robot arm 201. The drive mechanism 203 can be realized, for example, using the functions of the drive mechanism 203 illustrated in FIG. 1. The identification unit 4021 identifies the type of object based on image processing of an image of the object. The identification unit 4021 can be realized, for example, using the functions of the identification unit 4021 illustrated in FIG. 4. If the identification unit 4021 cannot identify the type of object, the change unit 4023 changes the environment in which the image of the target object was captured to a different environment. The change unit 4023 can be realized, for example, by using the function of the change unit 4023 illustrated in FIG.

[0049] Next, a description will be given of the processing of the robot system 1 with the minimum configuration. Fig. 10 is a diagram showing an example of a processing flow of the robot system 1 with the minimum configuration. Here, the processing of the robot system 1 with the minimum configuration will be described with reference to Fig. 10.

[0050] The robot arm 201 is capable of grasping an object including a transparent packaging material and an object packaged in the packaging material. The drive mechanism 203 drives the robot arm 201 (step S101). The identification unit 4021 identifies the type of object based on image processing of the image of the object (step S102). If the identification unit 4021 cannot identify the type of object, the change unit 4023 changes the environment in which the image of the object was captured to a different environment (step S103). In this way, the robot system 1 can change the environment in which an image of an object packaged in a transparent packaging material is captured to an environment in which an image that identifies the object can be captured. As a result, the robot system 1 can appropriately and accurately recognize the object even if the object is packaged in a transparent packaging material.

[0051] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0052] Although the embodiments of the present disclosure have been described, the robot system 1, the robot 20, the processing device 40, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0053] 11 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 11 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the robot system 1, the robot 20, the processing device 40, and other control devices described above are each implemented in the computer 5. The operations of each of the processing units described above are stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0054] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if the program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0055] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0056] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0057] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0058] (Supplementary Note 1) A robot system comprising: a robot arm capable of grasping an object including a transparent packaging material and an object packaged in the packaging material; a driving means for driving the robot arm; an identifying means for identifying the type of the object based on image processing of an image of the object; and a changing means for changing the environment in which the image of the object was taken to a different environment when the identifying means is unable to identify the type of the object.

[0059] (Supplementary Note 2) The robot system according to Supplementary Note 1, further comprising: an imaging means capable of capturing an image of the object; and the changing means, when the identifying means is unable to identify the type of object, controls the imaging means so that the angle of the imaging means is different from the angle of the imaging means that captured the image of the object.

[0060] (Supplementary Note 3) The robot system according to Supplementary Note 1, further comprising: an illumination means for illuminating the object; and the change means, if the identification means is unable to identify the type of object, changes the state of light irradiated on the object to a state different from the state of light irradiated on the object when the image of the object was captured.

[0061] (Supplementary Note 4) The robot system according to Supplementary Note 3, wherein the change means controls the lighting means so that the angle of the light is different from the angle of the light when the image of the target object was captured, if the identification means is unable to identify the type of the object.

[0062] (Supplementary Note 5) The robot system according to Supplementary Note 3, wherein the change means moves an object that changes the refractive index of light between the target object and the lighting means when the identification means is unable to identify the type of the object.

[0063] (Supplementary Note 6) The robot system according to Supplementary Note 5, further comprising: a second robot arm separate from the robot arm; and the change means, when the identification means is unable to identify the type of the object, moves the second robot arm between the target object and the illumination means.

[0064] (Supplementary Note 7) The robot system according to Supplementary Note 1, wherein the change unit controls the drive unit so as to change the state of the target object when the identification unit is unable to identify the type of the object.

[0065] (Supplementary Note 8) The robot system according to Supplementary Note 7, wherein the change unit controls the drive unit to change the orientation of the target object when the identification unit fails to identify the type of the object.

[0066] (Supplementary Note 9) The robot system according to Supplementary Note 7, wherein the change unit controls the drive unit so that the state of the packaging material changes when the identification unit fails to identify the type of the object.

[0067] (Supplementary Note 10) The robot system according to Supplementary Note 9, wherein the change unit controls the drive unit to suppress bulging of the packaging member when the identification unit fails to identify the type of the object.

[0068] (Supplementary Note 11) The robot system according to Supplementary Note 9, wherein the change unit controls the drive unit to stretch the packaging member when the identification unit fails to identify the type of the object.

[0069] (Supplementary Note 12) The robot system according to any one of Supplementary Note 1 to Supplementary Note 11, wherein the change means, if the identification means cannot identify the type of the object, changes the environment in which the image of the target object was captured to a different environment based on a trained model whose coefficients are determined by a supervised learning technique.

[0070] (Supplementary Note 13) The robot system according to Supplementary Note 12, wherein the change means changes the trained model based on the changed environment, and changes the environment from the environment in which the image of the object was captured to a different environment based on the changed trained model.

[0071] (Supplementary Note 14) The robot system described in any one of Supplementary Note 1 to Supplementary Note 11, wherein the change means performs a first processing content if the identification means is unable to identify the type of the object, and performs a second processing content different from the first processing content if the identification means is unable to identify the type of the object based on an image of the target object taken after the first processing content is performed.

[0072] (Supplementary Note 15) A processing method executed by a robot system having a robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material, the processing method comprising: driving the robot arm; identifying a type of the object based on image processing of an image of the object; and, if the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment.

[0073] (Supplementary Note 16) A recording medium storing a program that causes a computer of a robot system equipped with a robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material to: drive the robot arm; identify the type of the object based on image processing of an image of the object; and, if the type of the object cannot be identified, change the environment in which the image of the object was taken to a different environment.

[0074] (Supplementary Note 17) A robot system comprising: a robot arm; and a control means for controlling the operation of the robot arm so that the robot arm performs an action on an object based on a recognition result of an image of the object photographed by an imaging device, wherein the object is an object packaged in a transparent packaging material, and when the control means cannot recognize the object from the image, the control means controls the robot arm to change an environment in which the imaging device photographs the object.

[0075] According to each aspect of the present disclosure, even if an object (e.g., a product) is packaged in a packaging material that has transparency, the object can be appropriately and accurately recognized.

[0076] REFERENCE SIGNS LIST 1 robot system 5 computer 6 CPU 7 main memory 8 storage 9 interface 10 transport device 20 robot 30 lighting device 40 processing device 101 transport mechanism 201 robot arm 202 imaging device 203 driving mechanism 401 acquisition unit 402 processing unit DB database T tray TBL data table

Claims

1. A robot arm capable of grasping an object including a transparent packaging material and an object to be packaged by the packaging material; A driving means for driving the robot arm; an identification means for identifying the type of the object based on image processing of an image of the object; a change means for changing an environment in which an image of the object was captured to a different environment when the identification means is unable to identify the type of the object; A robot system comprising:

2. An imaging means capable of capturing an image of the object; Equipped with The change means is If the identification means cannot identify the type of the object, the imaging means is controlled to be at an angle different from the angle of the imaging means used to capture the image of the object. The robot system of claim 1 .

3. Illumination means for illuminating the object; Equipped with The change means is If the identification means cannot identify the type of the object, the light condition is changed so that the light condition irradiated on the object is different from the light condition when the image of the object was captured. The robot system of claim 1 .

4. The change means is If the identification means cannot identify the type of the object, the illumination means is controlled so that the angle of the light is different from the angle of the light when the image of the object was captured. The robot system according to claim 3 .

5. The change means is if the identification means cannot identify the type of the object, an object that changes the refractive index of light is moved between the target object and the illumination means; The robot system according to claim 3 .

6. a second robotic arm separate from the robotic arm; Equipped with The change means is if the identification means cannot identify the type of the object, moving the second robot arm between the target object and the lighting means; The robot system according to claim 5 .

7. The change means is If the identification means cannot identify the type of the object, the drive means is controlled so as to change the orientation of the object. The robot system of claim 1 .

8. The change means is If the identification means is unable to identify the type of the object, a first processing content is carried out, and if the identification means is unable to identify the type of the object based on an image of the target object captured after the first processing content is carried out, a second processing content different from the first processing content is carried out. The robot system according to any one of claims 1 to 7.

9. A processing method executed by a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, comprising: Driving the robot arm; identifying a type of the object based on image processing of an image of the object; If the type of the object cannot be identified, the environment in which the image of the object was taken is changed to a different environment. Processing methods.

10. A computer included in a robot system including a robot arm capable of grasping an object including a transparent packaging member and an object to be packaged by the packaging member, Driving the robot arm; identifying the type of object based on image processing of the image of the object; if the type of the object cannot be identified, changing the environment in which the image of the object was taken to a different environment; A program that executes the following.