Control device, handling system, determining device, control method and computer program

Abstract

According to one embodiment, a control device (3; 3-2) recognizes an object area from image information that corresponds to an object as a target in automated handling operations. The control device generates a handling strategy for the object and outputs a speech representation of the handling strategy. The control device constructs the object area based on a speech representation of an object shape encompassed in the handling strategy. The control device evaluates the handleability of the object based on geometric information about the object area. The control device evaluates the adaptability of a handling tool (14) to the object based on the speech representation of the object shape.Based on the manageability and adaptability of the handling tool (14), the control device determines whether handling of the object is possible and / or generates a handling position and a posture of the handling tool (14).

Description

AREA The embodiments described here generally refer to a control device, a handling system, a determining device, a control method and a computer program. BACKGROUND There is a conventional robot system that automates the handling of objects, for example an automated picking system that handles packages loaded in large quantities in a logistics warehouse. Such a robot system automatically calculates a gripping position and posture for an object, as well as a packaging position and posture at an input point, based on sensor data such as image information, whereupon a robot with a gripping mechanism carries out the gripping and packaging in practice. There is a need for a motion generation technology for a robot that can handle different objects under different environmental conditions. There is also a need for a technology that can determine whether a robot can properly follow the movement generated for it. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows an exemplary configuration of a handling system in a first embodiment; Fig. 2A shows example 1 of a removal tool unit in the first embodiment; Fig. 2B shows example 2 of the removal tool unit in the first embodiment; Fig. 2C shows example 3 of the removal tool unit in the first embodiment; Fig. 2D shows example 4 of the removal tool unit in the first embodiment; Fig. 3 is a table showing exemplary information on the gripping tool in the first embodiment; Fig. 4A shows an exemplary functional configuration of a control unit in the first embodiment; Fig. 4B is a flowchart describing the operation of the control unit in the first embodiment; Fig. 5A is an explanatory diagram for example 1 of the processing of a recognition unit in the first embodiment; Fig.Fig. 5B is an explanatory diagram for Example 2 of the processing of the recognition unit in the first embodiment; Fig. 5C is an explanatory diagram for Example 3 of the processing of the recognition unit in the first embodiment; Fig. 5D illustrates an exemplary functional configuration of a prompt generation unit in the first embodiment; Fig. 6A is a flowchart describing Example 1 of a prompt generation method in the first embodiment; Fig. 6B is an explanatory diagram for the processing in steps S1 and S2 in Fig. 6A; Fig. 6C shows an exemplary image after the processing in steps S1 and S2 in Fig. 6A; Fig. 7 shows an exemplary processing of the range determination in the first embodiment; Fig. 8A shows Example 1 for the generation of candidate points in the first embodiment; Fig. 8B shows Example 2 for the generation of candidate points in the first embodiment; Fig.Fig. 8C shows Example 3 for generating candidate points in the first embodiment; Fig. 8D shows Example 4 for generating candidate points in the first embodiment; Fig. 8E shows Example 5 of generating candidate points in the first embodiment; Fig. 8F shows Example 6 of generating candidate points in the first embodiment; Fig. 9A shows Example 1 of calculating a normal vector in the first embodiment; Fig. 9B shows Example 2 of calculating a normal vector in the first embodiment; Fig. 10 illustrates an exemplary clustering processing in the first embodiment; Fig. 11A illustrates an exemplary calculation of a prompt in the first embodiment; Fig. 11B illustrates an exemplary calculation of a prompt in a case where there is no hole in an area specified by a cluster in the first embodiment; Fig.Figure 11C illustrates an exemplary computation of a prompt in a case where a hole is present in an area specified by a cluster in the first embodiment; Figure 12 illustrates an exemplary detection of a facial area in the first embodiment; Figure 13 is an explanatory diagram for Example 2 of the prompt generation method in the first embodiment; Figure 14 is an explanatory diagram for an exemplary processing of a strategy generation unit in the first embodiment; Figure 15A illustrates an exemplary speech representation for an editing strategy in the first embodiment; Figure 15B illustrates an exemplary output of the speech representation of Figure 15A; Figure 16A is a flowchart illustrating Example 1 of the processing of a reconstruction processor in the first embodiment; FigureFigure 16B is a flowchart illustrating Example 2 of the processing of the reconstruction processor in the first embodiment; Figure 16C is a flowchart illustrating Example 3 of the processing of the reconstruction processor in the first embodiment; Figure 17A illustrates Example 1 of the processing of an object feature evaluation unit in the first embodiment; Figure 17B illustrates Example 2 of the processing of the object feature evaluation unit in the first embodiment; Figure 17C illustrates Example 3 of the processing of the object feature evaluation unit in the first embodiment; Figure 17D illustrates Example 4 of the processing of the object feature evaluation unit in the first embodiment; Figure 18 shows an exemplary processing target area of ​​the object feature evaluation unit in the first embodiment; FigureFigure 19A shows an exemplary processing of a tool adaptability evaluation unit in the first embodiment; Figure 19B shows an exemplary processing target area of ​​the tool adaptability evaluation unit in the first embodiment; Figure 20A shows an exemplary processing of an integration unit in the first embodiment; Figure 20B shows an exemplary processing target area of ​​the integration unit in the first embodiment; Figure 21A shows example 1 of the functional configuration of a control unit in a second embodiment; Figure 21B shows example 2 of the functional configuration of the control unit in the second embodiment; and Figure 22 shows an exemplary hardware configuration of the control unit in the first or second embodiment. DETAILED DESCRIPTION Embodiments of a control device, a handling system, a determining device, a control method and a program are described in detail below with reference to the accompanying drawings. First embodiment First, an overview of a handling system is given, which includes a handling device (picking robot) as an example of an object handling robot and an integrated robot management system. General overview Fig. 1 is a schematic diagram of the configuration of a handling system 100 in a first embodiment. The handling system 100 in the first embodiment comprises a handling device (a manipulator 1, a housing 2 and a controller 3), a sensor holder 4, an item container sensor 5, a sensor 6 for measuring the gripped item, a collection container sensor 7, a sensor 8 for temporary storage, an item container guide 9, an item container scale 10, a collection container guide 11 and a collection container scale 12. The sensor holder 4 holds sensors (the item container sensor 5, the sensor 6 for measuring the picked item, the collection container sensor 7 and the sensor 8 for intermediate storage). The item container sensor 5 measures the internal condition of an item container 101. The item container sensor 5 is, for example, an image sensor that is installed above the item container guide 9. The sensor 6 for measuring the grasped item is installed near the item container sensor 5 and measures the item grasped by the manipulator 1. The collection container sensor 7 measures the internal condition of a collection container 102. The collection container sensor 7 is, for example, an image sensor that is installed above the collection container guide 11. The intermediate storage sensor 8 measures an item that is temporarily stored in an intermediate storage unit 103. The item container guide 9 guides the item container 101, in which an item is stored as a handling target. The item container scale 10 measures the weight of item container 101. The collection container guide 11 directs a collection container 102 to collect the article picked up by the manipulator 1. The collection container scale 12 measures the weight of the collection container 102. It should be noted that the item container sensor 5, the sensor 6 for measuring the picked item, the collection container sensor 7, and the sensor 8 for temporary storage can be freely selectable sensors. Sensors are used that can capture image and three-dimensional information, such as an RGB camera, a depth camera, a laser rangefinder, and a light detection and distance measuring device / laser image detection and distance measuring device (LiDAR). It should be noted that the handling system 100 in the first embodiment further comprises a drive unit for operating various types of sensors and various types of drivers, a cylinder for storing compressed air, a compressor, a vacuum pump, a control unit, an external interface such as a user interface (UI) and a safety mechanism including a light curtain and a collision detector, which are not shown in the schematic representation of Fig. 1. The manipulator 1 comprises an arm 16 and a gripping tool unit 14. The arm 16 serves as an articulated robot driven by several servo motors. The arm 16 comprises a vertical articulated robot with six axes (axes 13a, 13b, 13c, 13d, 13e and 13f), which is shown in Fig. 1 as a representative vertical articulated robot with axes, a SCARA robot and a linear motion robot in combination. The gripping tool unit (handling tool) 14 has a mechanism that implements gripping a target, for example a suction, clamping, pinching or multi-finger mechanism. Figures 2A to 2D show examples 1 to 4 of the gripping tool unit 14 in the first embodiment. As shown in Figures 2A to 2D, a drive unit or various sensors for bending and rotation can be provided as the end effector for the handling device. In Example 1 in Fig. 2A, the gripping tool unit 14 comprises a force sensor 21, a bending shaft 22, and suction cups 23. In Example 2 in Fig. 2B or in Example 3 in Fig. 2C, the gripping tool unit 14 comprises a force sensor 21 and a suction cup / suction cups 23. In Example 4 in Fig. 2D, the gripping tool unit 14 comprises a force sensor 21 and a clamping mechanism 24. The gripping tool units 14 shown in Figs. 2A to 2C are of the suction type, but differ in the number of suction cups, their shape, their position, etc. As shown in Fig. 2D, the gripping tool unit 14 can have a clamping or multi-finger mechanism instead of a suction type. Fig. 3 is a table showing exemplary gripping tool information in the first embodiment. The gripping tool information in the first embodiment includes the gripping tool number, type, joint information, number of suction cups, information on the suction cups, number of clamping fingers, information on the clamping fingers, external shape information, and characteristic information. The gripper number is an identifier for each gripper unit 14. The type specifies the type of gripper unit 14. The joint information is information about a joint provided in the gripper unit 14. The number of suction cups refers to the number of suction cups 23 of the gripper unit 14. The suction cup information is information about the suction cups 23 provided in the gripper unit 14. The number of gripper fingers is the number of fingers of a gripping mechanism provided in the gripper unit 14. The gripper finger information is information about the fingers of the gripping mechanism provided in the gripper unit 14. The external shape information is information about the external shape of the gripper unit 14. The characteristic information is information about the properties of the gripper unit 14. Referring to Fig. 1, a robot-integrated management system 15 is used to manage the handling system 100. The gripper tool information in Fig. 3 is stored in a database of the robot-integrated management system 15. The gripper tool unit 14 can be detachably attached to the arm 16 via a gripper tool changer. The gripper tool unit 14 can be replaced by another gripper tool unit 14 according to an instruction from the robot-integrated management system 15. Exemplary Functional Configuration Fig. 4A shows an exemplary functional configuration of the control unit 3 in the first embodiment. The control unit 3 in the first embodiment comprises a recognition unit 31, a strategy generation unit 32, a reconstruction processor 33, an object feature evaluation unit 34, a tool adaptability evaluation unit 35, an integration unit 36, a display control unit 37, and a storage unit 38. Fig. 4B is a flowchart describing the operation of the control unit 3 in the first embodiment. The recognition unit 31 generates an object area from an image, and the strategy generation unit 32 generates a handling strategy from the image. The handling strategy relates, for example, to an area of ​​interest, a basic shape, an arrangement, and the placement of an object for successful handling. In a case where the detection unit 31 successfully performs the detection (step S51, Yes), specifically in a case where an object is present and the object is detected, the reconstruction processor 33 extracts a point cloud from the object area and fits the point cloud to the basic shape of the object to reconstruct the point cloud. The object feature evaluation unit 34 evaluates the feature of the object in the reconstructed point cloud within the area of ​​interest. Subsequently, the object feature evaluation unit 34 generates an affordance map from the geometric feature of the object, indicating the ease of handling of the object. Simultaneously, the tool adaptability evaluation unit 35 assesses whether a tool can come into contact with the object in an orientation opposite to the orientation of the point. The geometric property is determined, for example, by the shape (e.g., sphere, polyhedron, circle, or a polygon with three or more angles), the size (e.g., dimensions or area) of the object, the object's position in space (e.g., line symmetry or point symmetry), the object's location in a specific space (e.g., object position and orientation, relationship between neighboring objects or obstacles), etc. Next, the integration unit 36 ​​integrates the affordance map and a contactable area and calculates a manageable area. It should be noted that the term "manageable" as used in this description refers to a state in which the gripping tool unit 14 can perform stable handling (e.g., gripping) of an object. The integration unit 36 ​​evaluates the size of the manageable area. If the size is greater than a predefined threshold (step S52, Yes), the integration unit 36 ​​determines "manageable" (handling OK). If the size is equal to or less than the threshold (step S52, No), the integration unit 36 ​​determines "handling NG". The storage unit 38 stores "handling OK / NG" in a database and also stores the handling position and posture for "handling OK". Referring to Fig. 4A, the recognition unit 31 detects an area of ​​an object or an area of ​​a surface of the object. The strategy generation unit 32 outputs a handling strategy for the object in language form, taking into account domain knowledge about its handling. Domain knowledge refers, for example, to the experience, knowledge, or information a person gains through working with an object. The handling strategy takes into account complex elements such as the work environment, the properties of the object, the physical limitations of the robot, and the characteristics of the task. Furthermore, the handling strategy relates to the processing or movement required for the task. For example, the handling strategy can be a strategy for selecting the optimal processing method from among various processing methods. It can also be a strategy for determining the optimal procedure for the task during motion planning for the robot. Finally, the handling strategy can be a strategy for determining the optimal position and posture for the robot hand during motion generation, based on the work environment and the shape of the object. Based on an output from the strategy generation unit 32, the reconstruction processor 33 (an example of the construction processing unit) processes sensor information (e.g., image information including the object and point cloud information about the object) to improve the quality of the sensor information. The object feature evaluation unit 34 evaluates the ease of use (affordance value) based on the geometric features of each point on the object or on a surface of the object, or the geometric features of each point on the object in the area specified by the strategy generation unit 32. The tool adaptability evaluation unit 35 projects the shape of the tool from above onto the surface of the object from any point in the entire area of ​​the object or a specific surface of the object or in the area specified by the strategy generation unit 32, thus calculating an area in which the tool can come into contact with the object. Integration Unit 36 ​​calculates the average affordance value of the points on the object within the contactable area calculated by Tool Adaptability Evaluation Unit 35 and then defines a handleable area larger than a threshold value. Based on a position indicating the maximum affordance value, Integration Unit 36 ​​outputs the optimal handling position and posture. Furthermore, Integration Unit 36 ​​determines whether handling is feasible based on the size of the handleable area. Integration Unit 36 ​​then stores a set of information, including the optimal handling position and posture, a result indicating whether handling is feasible, and information about the object, in the database of Storage Unit 38. Details of the Recognition Unit: Figures 5A to 5C are explanatory diagrams for Examples 1 to 3 of the processing of the recognition unit 31 in the first embodiment. The recognition unit 31 has the function of recognizing the area of ​​an object detected in the image from sensor information, such as a color image, a depth image, or a point cloud. Examples 1 to 3 of a method for implementing the function of the recognition unit 31 are described with reference to Figures 5A to 5C. Example 1 in Fig. 5A shows a case in which an image processor 311 implements the function of the recognition unit 31. The image processor 311 recognizes an object area from an image using a predefined image processing algorithm (e.g., an image processing function in OpenCV). Example 2 in Fig. 5B shows a case in which a first machine learning processor 312 implements the function of the recognition unit 31. The first machine learning processor 312 recognizes an object region from an image using a machine learning model (e.g., a deep neural network). Furthermore, a procedure for collecting data for a new article and retraining a neural network with the collected data to further improve accuracy is described using Figure 5C. The description using Figure 5C is based on an exemplary zero-shot learning procedure. Example 3 in Fig. 5C shows a case in which a prompt generation unit 313 and a second machine learning processor 314 implement the function of the recognition unit 31. In Example 3 in Fig. 5C, the prompt generation unit 313 generates a prompt from image information. The prompt refers to input information used to instruct a machine learning model to output an expected response. There are various ways to represent such a prompt, such as speech, a dot, or a bounding box. The prompt generation unit 313 generates a prompt from image information, which represents an instruction for recognizing the area of ​​an item in a storage box or the area of ​​a surface of the item. In Example 3 in Fig. 5C, the image information and the prompt with position information for at least one point in an object area (asterisk in Fig. 5C), which are output by the prompt generation unit 313, are input into the second machine learning processor 314. From the image information and the input prompt, the second machine learning processor 314 recognizes the object area, including the position specified by the input prompt, by processing the machine learning model based on the input prompt. Since the input prompt has a significant influence on the machine learning output, a method for generating a suitable input prompt is required. Fig. 5D shows an exemplary functional configuration of the prompt generation unit 313 in the first embodiment. The prompt generation unit 313 comprises a third machine learning processor 3131, a determination unit 3132, and a generation unit 3133. The third machine learning processor 3131 calculates an approximate area of ​​an object. The determination unit 3132 evaluates the accuracy of the area and excludes an incorrect area (an area outside the object). The generation unit 3133 generates a candidate prompt (e.g., a dot) within the area. Next, an exemplary procedure for prompt generation by the prompt generation unit 313 is described. Fig. 6A is a flowchart describing Example 1 of a prompt generation method in the first embodiment. In Example 1 of Fig. 6A, an RGB-D image containing both depth data (D) and color data (RGB) is input as image information. First, the third learning processor, 3131, calculates an approximate area from an image (step S1). This calculation can be performed, for example, by a baseline image recognition model. The baseline model refers to a machine learning model that has been trained on a large, required dataset (i.e., pre-training). Next, the third machine learning processor 3131 excludes a background area (step S2). Fig. 6B is an explanatory diagram for the processing in steps S1 and S2 in Fig. 6A. The third machine learning processor 3131 calculates from the image an approximate area (e.g., a box area 211) within which an object is to be contained. Then, the third machine learning processor 3131 excludes, for example, a background area 212 on part of the storage box. Fig. 6C shows an example image after processing in steps S1 and S2 in Fig. 6A. In the example of Fig. 6C, the background area 212, which is present in the image in Fig. 6B, is deleted. The rough area calculated in step S1 is not necessarily correct. For example, a trademark or barcode on the object may be mistakenly identified as a different object. Therefore, the determination unit 3132 evaluates the accuracy of the area to perform an area determination to eliminate a false area (misidentified area) (step S3). Figure 7 illustrates an exemplary processing of the area determination in the first embodiment. The third learning processor 3131 cuts out an image of an area that is contained within the area of ​​a specific object and then inputs the image and the category of the area together into the determination unit 3132. The determination unit 3132 evaluates the degree of similarity between the category of the object and the cut-out image and calculates a score. If the score is lower than a threshold, the determination unit 3132 determines that the degree of similarity between the image and the category is low and then excludes the area of ​​the image as an inaccurate area. The processing of the determination unit 3132 can be implemented by a determiner based on a learning model that evaluates the degree of similarity between an image and a language. An object area after exclusion is the entire area of ​​the object. The area of ​​an object's surface can be recognized at the time of handling. Thus, the generation unit 3133 creates a prompt, and the second learning processor 314 also recognizes the area of ​​the object's surface. Initially, the generation unit 3133 generates candidate points across the entire area of ​​the object (step S4). Figures 8A to 8F illustrate Examples 1 to 6 of the generation of candidate points in the first embodiment. Candidate points can be generated at regular intervals or using a scanning-based technique. Next, the generating unit 3133 calculates the respective geometric features of the candidate points, for example their respective normal vectors (step S5). Figures 9A and 9B illustrate Examples 1 and 2 of the calculation of normal vectors in the first embodiment. The direction of each normal vector calculated by the generating unit 3133 is used to cluster the candidate points. Next, the generating unit 3133 performs a clustering of the candidate points based on the geometric features calculated in step S5 (step S6). Fig. 10 illustrates an exemplary clustering process in the first embodiment. The generating unit 3133 classifies candidate points that are similar in their geometric features into the same cluster (or group). In the example of Fig. 10, candidate points whose normal vectors have a similar direction are classified into the same cluster. Next, the generating unit 3133 calculates the average value of the positions of the candidate points in the cluster to calculate the midpoint in the cluster, and then generates the midpoint as an input prompt (step S7). Fig. 11A illustrates an example calculation of an input prompt in the first embodiment. The generating unit 3133 generates as an input prompt the center point of the area of ​​candidate points that have been classified into the same cluster. It should be noted that there may be a hole (or opening) in the area of ​​candidate points that have been classified into the same cluster, so the calculated midpoint may lie outside the area. Fig. 11B shows an example of the computation of a prompt in a case where, in the first embodiment, there is no hole in an area specified by a cluster. Fig. 11C shows an example of the computation of a prompt in a case where, in the first embodiment, there is a hole in an area specified by a cluster. In a case where there is no hole in an area specified by a cluster, the generating unit 3133 prompts the center point of the area of ​​candidate points classified in the same cluster. In a case where, for example, a hole is present in an area specified by a cluster, the generating unit 3133 prompts the four points that are highest in descending order of their distance from the background. As described above, in the example shown in Fig. 6A, the prompt generation unit 313 classifies the points into at least one group based on their geometric features. These points represent a portion of the object's area (e.g., a surface area) or the entire object area. Based on the points contained in this group, it then generates a prompt that provides an instructional representation to facilitate the recognition of a face area, indicating a face of the object. The second machine learning processor 314 then recognizes the face area from the image information based on this prompt. Fig. 12 illustrates an exemplary detection of a surface area in the first embodiment. According to the second machine learning processor 314 in the first embodiment, for example, two surfaces of a box can be detected in addition to the entire box, as shown in Fig. 12. Fig. 13 is an explanatory diagram for Example 2 of the prompt generation method in the first embodiment. As shown in step S11 in Fig. 13, the prompt generation unit 313 can calculate a prompt from the image information using a machine learning model. In this case, training data for building the machine learning model used for processing in Fig. 13 is created using the method of Fig. 6A. Details of the Strategy Generation Unit: Fig. 14 is an explanatory diagram for an exemplary processing of the strategy generation unit 32 in the first embodiment. The strategy generation unit 32 generates a language representation that specifies a processing strategy for an object contained in an image. The processing strategy includes not only a strategy based on a predefined rule (predefined rule-based processing), but also a strategy based on domain knowledge about the processing, which is difficult to represent through rule-based processing. For example, an area of ​​interest 201 shown in Fig. 14 is defined based on a language representation for a strategy generated by the strategy generation unit 32. In particular, the function of the strategy generation unit 32 is implemented by a large language model. In one example, learning is performed using a contrastive speech-image pre-training model (CLIP) such that an image and a speech representation match for an editing strategy. The strategy generation unit 32 encodes an image and a speech representation for an editing strategy using a training model (e.g., a transformer) and evaluates the degree of similarity between two encoded representations. Subsequently, the strategy generation unit 32 trains the large language model to decrease the degree of similarity of a representation in which an image and an editing strategy do not match. It should be noted that such a representation, in which an image and an editing strategy do not match, is, for example, a representation of an editing strategy that is ineffective against an editing target object specified from the image (ineffective editing strategy). The function of the strategy generation unit 32 can also be implemented, for example, by a visual reasoning model or a picture-language model. Furthermore, a machine learning model can be trained for the strategy generation unit 32 by inputting a query containing image information and a desired handling strategy based on that image information into the machine learning model. This will cause the machine learning model to output a language representation for the desired handling strategy. Furthermore, the strategy generation unit 32 can, for example, generate a language representation for a handling strategy by using large natural language models, such as the transformer described above, the visual inference model and the picture language model, or by combining components in a large natural language model. Fig. 15A illustrates an exemplary language representation for a processing strategy in the first embodiment. In the example of Fig. 15A, the language representation for a handling strategy includes "Category", "Basic Shape", "Placement Type", "Position relative to a Container (Storage Box)", "Grasp Method (Handling Method)", "Area of ​​Interest", and "Reason for Interest" with respect to an object. The "category" and the "basic form" are used for processing sensor information. Using the "category" and the "basic form" for processing sensor information allows for more accurate processing of the sensor information. The "placement method" and the "position relative to the container" are used in processing to determine a picking sequence for multiple objects. Using the "placement method" and the "position relative to the container" in processing to determine a picking sequence allows for the determination of a more effective picking sequence. The "gripping method" is used in the processing to select a handling tool. By using the "gripping method" in the processing to select a handling tool, a more suitable handling tool can be chosen. The "area of ​​interest" and the "reason for interest" indicate a strategy that is difficult to represent using a predefined rule. For example, the "area of ​​interest" and the "reason for interest" specify the area a person grasps when picking up an object, in relation to the object's shape and how the object is to be positioned (information about the posture in which the object is to be placed). The area specified by the "area of ​​interest" and the "reason for interest" is based on domain knowledge when a person grasps an object as a handling target. This domain knowledge can be acquired by learning a large-scale natural language model (e.g., the CLIP model described above), which enables the creation of a handling strategy applicable to different objects.It can also be gained by analyzing human movements in an image or video. However, the expertise is difficult to define based on rules. Furthermore, the "area of ​​interest" and the "reason for interest" can be used to process the delimitation of a computational area based on the area of ​​interest. For example, using the "area of ​​interest" to process the delimitation of a computational area allows for a reduction in computation time. Fig. 15B shows an example output of the speech representation from Fig. 15A. The strategy generation unit 32 derives a speech representation for an editing strategy from an image in order to output, for example, the speech representation shown in Fig. 15B. Details of the reconstruction processor With reference to Fig. 4A, the reconstruction processor 33 processes image information or point cloud information obtained as sensor information from a sensor in order to reconstruct the sensor information (e.g. interpolation for missing values ​​or removal of noise). In particular, the reconstruction processor 33 extracts sensor information (e.g., a depth image with respect to each object or surface and a point cloud specifying each object or surface) from information about an image area (e.g., an object area or a surface area of ​​an object) output by the recognition unit 31. The reconstruction processor 33 adapts the sensor information to a basic shape. Fig. 16A is a flowchart illustrating Example 1 of the processing by the reconstruction processor 33 in the first embodiment. In the example shown in Fig. 16A, an area for defining a processing target (object area or surface area) is entered as Input 1. Additionally, processing target data (depth image or point cloud) is entered as Input 2. For example, as in steps S21 to S27, the reconstruction processor 33 performs fitting processing until a suitable basic shape is found, in order to determine whether a candidate basic shape is a good match or not. In the example shown in Fig. 16A, the plane fitting processing is performed in steps S21 and S22, the cylindrical fitting processing in steps S24 and S25, and the spherical fitting processing in steps S26 and S27, in that order. These processes can also be performed in parallel. If the matching capability between the sensor information and a basic shape is higher than a threshold value (step S22, S25 or S27, yes), the reconstruction processor 33 reconstructs the sensor information (e.g. the image or the point cloud) by an equation for the basic shape (step S23). Fig. 16B is a flowchart illustrating Example 2 of the processing by the reconstruction processor 33 in the first embodiment. In the example of Fig. 16B, a basic shape, which specifies the shape of an object, is also inputted as input 3. The reconstruction processor 33 performs adaptation processing (step S31) based on a language representation for an object shape, which is input as input information (basic shape, which in the example of Fig. 16B is input 3), and reconstructs sensor information (e.g., an image or a point cloud) (step S32). Specifically, based on the language representation for an object shape, the reconstruction processor 33 selects an equation that represents the shape of the object and reconstructs an object region using the equation. The basic shape in the language representation can be represented by a suitable three-dimensional shape in three-dimensional space. For example, a language representation such as "plane" can be represented by a three-dimensional shape such as "ax + by + cz + d = 0". Note that (a, b, c) represents a normal vector with respect to a plane. In the example of Fig. 16B, the processing of the reconstruction processor 33 is faster than in the case of Fig. 16A, since it is not necessary to determine which basic shape should be used for the adaptation. It should be noted that the basic shape, input 3, is derived from image information by a machine learning model. In the first embodiment, the basic shape, input 3, is contained in a language representation for a processing strategy, which is output by the strategy generation unit 32. Fig. 16C is a flowchart illustrating Example 3 of the processing by the reconstruction processor 33 in the first embodiment. In the example of Fig. 16C, the processing of the reconstruction of sensor information (e.g., an image or a point cloud) is performed by a machine learning model (step S41). The input and output information in Fig. 16C is similar to that in the case of Fig. 16B. As in the example of Fig. 16C, the processing by the reconstruction processor 33 can be implemented by a machine learning model. Details on the Object Feature Evaluation Unit: The Object Feature Evaluation Unit 34 evaluates the geometric feature of an object captured in an image. The object's geometric feature influences an affordance value that indicates the manipulability of the object captured in the image. The object feature (e.g., of an item) represents the feature of the object itself. Even if a handling tool is changed, the object's property does not change. Figures 17A to 17D illustrate examples 1 to 4 of the processing of the object characteristic evaluation unit 34 in the first embodiment. For example, Fig. 17A illustrates an example in which the object feature rating unit 34 evaluates the flatness of the object as a feature. Flatness indicates the adaptability between a surface of the object and a plane. A plane is easier to handle than a curved surface, so a point in an area of ​​higher flatness has a higher affordance value. For example, Fig. 17B shows an example where the object feature evaluation unit 34 evaluates the object's orientation as a feature of the object. The object feature evaluation unit 34 calculates a normal to each point that is to be encompassed within an object area (e.g., a normal to a point on a surface of the object) and evaluates an object angle, which is the angle between the normal to each point that is to be encompassed within the object area and the orientation of the bottom of a storage box (normal of the bottom). A flatter placement allows for easier handling, so a point in an area with a smaller object angle has a higher affordance value. For example, Fig. 17C shows an example where the object feature evaluation unit 34 evaluates the distance to the object's center point as a feature of the object. The object's center point (e.g., the center point on a face of the object) is easier to handle, and therefore a point in an area closer to the object's center has a higher affordance value. If there are holes in the center (e.g., in donuts), the distance can be considered as the distance to the object's edge. Points with large distances to all edges have a high affordance value. For example, Fig. 17D shows an example in which the object feature rating unit 34 rates the object's height as a feature of an object. An object in a higher position is easier to handle, and therefore a point in the area of ​​an object in a higher position has a higher affordance value. The four types of affordance values ​​described above in Figures 17A to 17D are independent of the handling tool. Therefore, these affordance values ​​do not change even if the handling tool is changed. These four types of affordance indicators are examples, and other affordance indicators can also be used. For example, an affordance indicator annotated based on a perspective of its influence on ease of use can be used to assess the ease of use of an object. Fig. 18 shows an exemplary processing target area of ​​the object feature evaluation unit 34 in the first embodiment. The object feature evaluation unit 34 can evaluate the ease of handling of an object in a target area that is specified by a language representation for an area of ​​interest encompassed in a handling strategy within an object area instead of the entire object area (in an area of ​​interest 201 in the example of Fig. 18). Details of the unit for evaluating the adaptability of the tool: The tool adaptability evaluation unit 35 calculates a contact area between the suction cup attached to the handling tool and an object. Fig. 19A illustrates an exemplary processing of the tool adaptability evaluation unit 35 in the first embodiment. In the example of Fig. 19A, the tool adaptability evaluation unit 35 calculates a normal to each point in a surface region of the object. Along the direction of the normal, the tool adaptability evaluation unit 35 offsets the position of the point. Then, with the offset point as the center point of the suction cup shape 202, the tool adaptability evaluation unit 35 projects along a direction opposite to the direction of the normal, thereby obtaining a contact area 203 on the surface of the object where the suction cup shape makes contact with the object. Such projection processing is implemented, for example, by ray casting. Ray casting is a method for calculating a point where a ray emitted in the opposite direction to the normal direction from the offset point intersects a mesh or point cloud representing the object. Fig. 19B illustrates an exemplary processing target area of ​​the tool adaptability evaluation unit 35 of the tool in the first embodiment. The tool adaptability evaluation unit 35 can evaluate the adaptability of the handling tool in a target area within an object area specified by a language representation for an area of ​​interest encompassed in a handling strategy, instead of the entire object area (in an area of ​​interest 201 in the example of Fig. 19B). In this case, the tool adaptability evaluation unit 35 projects the outer shape of the handling tool onto a target area to compute a contact area 203 in which the handling tool comes into contact with the object and evaluates the adaptability of the handling tool based on the contact area 203. It should be noted that in a case where the gripping tool unit 14 includes a gripping hand or a multi-fingered hand, the tool adaptability evaluation unit 35 evaluates the adaptability of the tool taking into account a space into which the fingers can be inserted, in addition to an area where the fingers can come into contact with the object. If the gripping tool unit 14 performs a clamping action, the tool adaptability evaluation unit 35 evaluates the adaptability of the tool using a similar procedure to that used in the case of suction. Details of the integration unit: The integration unit 36 ​​integrates an evaluation result from the object characteristic evaluation unit 34 and an evaluation result from the tool adaptability evaluation unit 35 and calculates a manageable range to determine whether handling is possible or not. Fig. 20A illustrates an exemplary processing of the integration unit 36 ​​in the first embodiment. In the example of Fig. 20A, the integration unit 36 ​​calculates the average of the affordance values ​​evaluated by the object feature evaluation unit 34 at points in the contact region 203 at the time of contact between the suction cup and the object, as calculated by the tool adaptability evaluation unit 35. Next, in a case where the average value is greater than a threshold, the integration unit 36 ​​defines a range encompassing the points used in calculating the average value as the manageable range 204. In a case where the size of the manageable area 204 is greater than a threshold value, the integration unit 36 ​​determines that gripping is enabled. In a case where gripping is enabled, the integration unit 36 ​​also outputs the handling position and posture of the handling tool. Fig. 20B illustrates an exemplary processing target area of ​​the integration unit 36 ​​in the first embodiment. The integration unit 36 ​​can compute a manageable area 204 in a target area within an object area, which is specified by a language representation for an area of ​​interest encompassed in a handling strategy instead of the entire object area (in an area of ​​interest 201 in the example of Fig. 20B). The area of ​​interest 201 is calculated according to a handling strategy that incorporates domain knowledge about handling, acquired through machine learning of the large language model used by the strategy generation unit 32. Domain knowledge about handling refers, for example, to the experience, knowledge, or information a person gains by working with an object. Furthermore, the handling strategy considers elements such as the work environment, the properties of the object, the physical limitations of the robot, and the characteristics of the task in a complex manner. The use of the term "area of ​​interest" or "reason for interest" in the handling strategy allows for the narrowing down of the manageable area 204 as the most suitable area from among many manageable areas.Such a limitation allows for a reduction in the calculation time required for calculating the manageable area 204. Details of the Display Control Unit: Referring to Fig. 4A, the display control unit 37 displays at least one of the following information on a display device: output information from the strategy generation unit 32 described above, output information from the object feature evaluation unit 34 described above, output information from the tool adaptability evaluation unit 35 described above, or output information from the integration unit 36 ​​described above. The display control unit 37 allows an administrator or similar person to review an intermediate result of the control processing of the control unit 3. For example, the information output by the display control unit 37 on the display device can be used for debugging the control processing of the control unit 3. As described above, in the first embodiment (in an exemplary control device that controls automated handling operations), the recognition unit 31 in the control unit 3 recognizes an object area that specifies the first object from image information that includes a first object as the handling target. The strategy generation unit 32 generates a speech representation from the image information that corresponds to a handling strategy for the first object. The reconstruction processor 33 (an exemplary construction processor that constructs an object area) reconstructs the object area based on a speech representation for an object shape included in the handling strategy. The object feature evaluation unit 34 evaluates the handling-friendliness of the first object based on geometric information about the reconstructed object area.The tool adaptability assessment unit 35 evaluates the adaptability of a handling tool for the first object based on the language representation for the object shape. Then, the integration unit 36 ​​determines whether the handling of the first object is activated or not, and / or generates the handling position and posture of the handling tool based on the handling tool's ease of use and adaptability. Thus, according to the control unit 3 in the first embodiment, the handling position and posture of the handling tool, which handles various objects, can be controlled more precisely. In a conventional case, for example, a representation of a posture that enables grasping in an image is often estimated by a neural network based on image information (e.g., the image coordinates of a grasping position and the angle between a grasping posture and the image's horizontal axis). However, directly applying such a conventional neural network to grasping an unrecognized object leads to a deterioration in accuracy. In the prior art, the neural network must be retrained using data from unrecognized new objects. In the current state of the art, a physical simulator based on a dynamic model is used to assess whether a robot's handling tool can successfully make contact with an object. Changing the handling tool requires a reassessment, which is time-consuming. Furthermore, in the prior art, the movement of the handling tool is generated based on a rule. Domain knowledge about the handling, which is difficult to determine using a rule, is not taken into account. Therefore, generating a movement based on a handling strategy similar to that of a human cannot be implemented. In contrast to the prior art, the prompt generation unit 313 in the first embodiment of the control unit 3 described above (exemplary control device) generates a prompt to instruct a neural network to recognize an unrecognized object. Therefore, a handling movement can be generated for such an unrecognized object, allowing the control unit 3 in the first embodiment to be applied to various article shapes. Furthermore, in the first embodiment described above, the object feature evaluation unit 34 evaluates the feature of an object, and the tool adaptability evaluation unit 35 evaluates the adaptability of the handling tool. With this configuration, the object feature and the adaptability of the handling tool are evaluated independently of each other. Therefore, even if the handling tool is changed, only the object feature and adaptability of the handling tool need to be re-evaluated. The decision as to whether handling is possible or not can be made more quickly than before. Furthermore, in the first embodiment of the controller 3 described above, the strategy generation unit 32 uses a large language model to calculate a language representation that contains domain knowledge about handling in order to create a handling strategy that is similar to that of a human. Therefore, the accuracy of the handling movement can be further improved compared to before. Furthermore, in the first embodiment described above, the reconstruction processor 33, the object feature evaluation unit 34, the tool adaptability evaluation unit 35, and the integration unit 36 ​​in the control unit 3 use the language representation (e.g., "region of interest") for the handling strategy, which was obtained from the large language model of the strategy generation unit 32. Therefore, the accuracy and speed of processing can be further improved compared to before. It should be noted that a function for determining the ease of handling of an object during automated handling operations, which is included in the function of control unit 3 in the first embodiment, can be implemented as a determination device. Such a determination device comprises, for example: a recognition unit that recognizes an object area representing the object from image information that includes the object as the handling target; a strategy generation unit that generates a language representation from the image information that corresponds to a handling strategy for the object; and a reconstruction processor that constructs the object area based on a language representation for an object shape included in the handling strategy. The determination device then determines the ease of handling of the object based on geometric information about the constructed object area. Second embodiment Next, a second embodiment is described. Similarities between the second embodiment and the first embodiment are not described; instead, the differences between the second and first embodiments are described. The second embodiment includes, in addition to a function identical to that in the first embodiment, a function for redirecting the previous determination result (whether handling is activated and the handling position and posture). Exemplary functional configuration Fig. 21A shows example 1 of the functional configuration of a control unit 3-2 in the second embodiment. In addition to a configuration identical to that in the first embodiment (Fig. 4A), the control unit 3-2 in the second embodiment comprises an extraction unit 41 and a search unit 42. It should be noted that a recognition unit 31, a strategy generation unit 32, a reconstruction processor 33, an object feature evaluation unit 34, a tool adaptation capability evaluation unit 35, an integration unit 36 ​​and a display control unit 37 are omitted in Fig. 21A. A storage unit 38 stores the past determination result. The past determination result includes, for example, the characteristic of an object determined in the past, whether the handling of the object is activated or not, as well as the handling position and posture of the object. Extraction unit 41 extracts the object's feature from an image of the object. Such an image of an object could, for example, be an image of a new item that has not previously been determined by control unit 3-2. Extraction unit 41 is implemented, for example, by a machine learning model. Examples of such a machine learning model include, in particular, a deep convolutional neural network and a transformer. The search unit 42 reads the previously determined feature of an object from the memory unit 38 and calculates the degree of similarity with the input feature of a new object. The search unit 42 then outputs information on whether the handling of an object whose feature similarity is greater than a threshold is activated, as well as information on the handling position and posture. It should be noted that the input information (e.g., information about a new article) to extraction unit 41 can be text data instead of an image. Fig. 21B illustrates Example 2 of the functional configuration of the control unit 3-2 in the second embodiment. Fig. 21B illustrates a case in which information about a new item is entered as text information. In this case, the storage unit 38 stores the characteristic of an object as text information. The extraction unit 41 extracts the characteristic of an object from the text information. Subsequently, the search unit 42 reads text information from the storage unit 38 that specifies the previously determined characteristic of an object and calculates the degree of similarity with the entered text information that specifies a new object. As described above, in the second embodiment, the extraction unit 41 in the control unit 3-2 (exemplary control device) extracts the feature of the first object based on image information encompassing a first object and / or based on text information specifying the feature of the first object. The search unit 42 searches a storage device that stores a second feature of a second object to determine whether handling of the second object is enabled and to ascertain the handling position and attitude of a handling tool that handles the second object. Subsequently, the search unit 42 outputs information on whether handling of the second object is enabled with the second feature whose degree of similarity to a first feature is greater than a threshold, as well as information on the handling position and attitude of the handling tool that handles the second object. Thus, according to the control unit 3-2 in the second embodiment, the past determination result stored in the storage unit 38 can be redirected. For example, in a case where a new item is similar to a previously determined item, the determination of whether handling is activated or not, and the determination of a handling movement (handling position and posture), can be carried out more quickly. Finally, an exemplary hardware configuration of controllers 3 and 3-2 in the first and second embodiments is described. The hardware configuration of controller 3 in the first embodiment is described below, but the same configuration can be applied to controller 3-2 in the second embodiment. Exemplary Hardware Configuration Fig. 22 shows an exemplary hardware configuration of the controller 3 in the first or second embodiment. The controller 3 comprises a processor unit 301, a main memory device 302, an auxiliary memory device 303, a display device 304, an input device 305, and a communication device 306. The processor unit 301, the main memory device 302, the auxiliary memory device 303, the display device 304, the input device 305, and the communication device 306 are interconnected via a bus 310. It should be noted that the display device 304, the input device 305, and the communication device 306 are not necessarily required. For example, if the control unit 3 is connected to another device, the display, input, and communication functions of that other device can be used. The processor unit 301 executes a computer program that is read from the auxiliary storage device 303 into the main memory device 302. The processor unit 301 comprises one or more processors, such as central processing units (CPUs). Examples of the main memory device 302 are read-only memory (ROM) and random-access memory (RAM). Examples of the auxiliary storage device 303 are a memory card and a hard disk drive (HDD). The display device 304 displays information. The display device 304 is, for example, a liquid crystal display. The input device 305 receives information as input. The input device 305 includes, for example, hardware buttons. It should be noted that the display device 304 and the input device 305 can be implemented, for example, by a liquid crystal touch panel with a display function and an input function. The communication device 306 communicates with another device. The program to be executed by Controller 3 is stored on a computer-readable storage medium, such as a CD-ROM, a memory card, a CD-R or a Digital Versatile Disk (DVD), in an installable file format or in an executable file format for delivery as a computer program product. The program to be executed by control unit 3 can be stored on a computer connected to a network, such as the internet, and made available by downloading it over the network. Alternatively, the program to be executed by control unit 3 can be made available via a network, such as the internet, instead of being downloaded. The program to be executed by control unit 3 may be pre-installed on a ROM or the like for provision. The program to be executed by control unit 3 has a module configuration that includes a function that can be implemented by the program and is included in the function of control unit 3. The processor unit 301 reads the program from a storage medium, such as the auxiliary storage device 303, and executes the program. As a result, the function implemented by the program is loaded onto the main memory device 302. In other words, the function implemented by the program is created on the main memory device 302. It should be noted that some of the functions of control unit 3 can be implemented by hardware, such as an integrated circuit (IC). The IC is, for example, a processor that performs specific processing. In a case where functions are implemented by multiple processors, each processor can implement one of the functions, or two or more functions from the set of functions. Although certain embodiments have been described, these embodiments serve only as examples and are not intended to limit the scope of the inventions. In fact, the new embodiments described herein can be embodied in a multitude of other forms; moreover, various omissions, substitutions, and modifications in the form of the embodiments described herein can be made without departing from the spirit of the inventions. The appended claims and their equivalents are intended to cover such forms or modifications that would fall within the scope and spirit of the inventions.

Claims

A control device (3; 3-2) comprising: a recognition unit (31) configured to recognize an object area corresponding to a first object from image information encompassing the first object, wherein the first object is a handling target in automated handling operations; a strategy generation unit (32) configured to generate a handling strategy for the first object based on the image information and output a speech representation of the handling strategy; a construction processing unit (33) configured to construct the object area based on a speech representation of an object shape encompassed in the handling strategy; an object feature evaluation unit (34) configured to evaluate the ease of handling of the first object based on geometric information about the constructed object area;a tool adaptability evaluation unit (35) configured to evaluate the adaptability of a handling tool (14) for the first object based on the language representation for the object shape; and an integration unit (36) configured to perform the following based on the ease of use and adaptability of the handling tool (14): determining whether or not handling the first object is possible, and / or generating a handling position and attitude of the handling tool (14). Control device (3;3-2) according to claim 1, wherein the recognition unit (31) is configured to determine whether an area in the image information displays a part of the first object or an object other than the first object, generates the object area that displays the first object while deleting an area of ​​the other object, classifies points that display a part of the generated object area into at least one group based on geometric information about the points, generates a prompt that represents an instruction to capture a face area of ​​the first object based on the points included in the at least one group, and captures the face area from the image information based on the prompt. Control device (3; 3-2) according to claim 1 or 2, wherein the object feature evaluation unit (34) is configured to perform the evaluation of the ease of handling of the first object in a target area within the object area, wherein the target area is specified by a language representation for an area of ​​interest included in the handling strategy. Control device (3; 3-2) according to one of claims 1 to 3, wherein the tool adaptability evaluation unit (35) is configured to perform the evaluation of the adaptability of the handling tool (14) in a target area in the object area, wherein the target area is specified by a language representation for an area of ​​interest included in the handling strategy. Control device (3; 3-2) according to claim 4, wherein the tool adaptability evaluation unit (35) is configured to calculate a contact area in which the handling tool comes into contact with the first object, wherein the contact area is projected by projecting an outer shape of the handling tool (14) onto the target area in the object area, and performs the evaluation of the adaptability of the handling tool (14) on the basis of the contact area. Control device (3; 3-2) according to claim 5, wherein the integration unit (36) is configured to determine, with the contact area as its target, whether or not the handling of the first object is possible, and / or to generate the handling position and the posture of the handling tool (14). Control device (3; 3-2) according to one of claims 1 to 6, wherein the design processing unit (33) is configured to select an equation that represents a shape of the first object based on the language representation for the object shape, and performs the design of the object area using the equation. Control device (3-2) according to one of claims 1 to 7, further comprising: an extraction unit (41) configured to extract a first feature of the first object based on image information and / or based on text information specifying a feature of the first object;and a search unit (42) configured to search a storage device that stores a second feature of a second object, information about whether or not the handling of the second object is enabled, and information about a handling position and attitude of a handling tool (14) with which the second object is handled, and outputs information about whether or not the handling of the second object with the second feature, whose degree of similarity to the first feature is greater than a threshold, is enabled, as well as information about the handling position and attitude of the handling tool (14) with which the second object is handled. Control device (3; 3-2) according to one of claims 1 to 8, further comprising a display control unit (37) configured to display at least one of the following information on a display device (304): output information from the strategy generation unit (32), output information from the object feature evaluation unit (34), output information from the tool adaptability evaluation unit (35) and output information from the integration unit (36). Control device (3; 3-2) according to one of claims 1 to 9, wherein the language representation comprises domain knowledge relating to the handling of the first object. A handling system comprising: the control device (3; 3-2) according to any one of claims 1 to 10; and a manipulator (1) with the handling tool (14), the handling position and holding of which is controlled by the control device (3; 3-2). A determination device comprising: a recognition unit (31) configured to recognize an object area corresponding to an object from image information encompassing the object, wherein the object is a handling target in automated handling operations; and a strategy generation unit (32) configured to generate a handling strategy for the object based on the image information and output a speech representation of the handling strategy; a construction processing unit (33) configured to construct the object area based on a speech representation of an object shape encompassed in the handling strategy, wherein the determination device is configured to determine the ease of handling of the object based on geometric information about the constructed object area. Determination device according to claim 12, wherein the language representation contains domain knowledge related to the handling of the object. A control procedure implemented by a computer, wherein the control procedure comprises: capturing an object area corresponding to an object from image information encompassing the object, the object being a handling target in automated handling operations; generating a handling strategy for the object based on the image information; outputting a speech representation of the handling strategy; constructing the object area based on a speech representation of an object shape encompassed in the handling strategy; evaluating the ease of handling the object based on geometric information about the constructed object area; and evaluating the adaptability of a handling tool (14) to the object based on the speech representation of the object shape.and performing a determination, based on the ease of use and adaptability of the handling tool (14), as to whether the handling of the object is enabled, and / or creating a handling position and a posture of the handling tool (14).; Control method according to claim 14, wherein the language representation includes domain knowledge related to the handling of the object. A computer program comprising programmed instructions that cause a computer to execute the control method according to claim 14. Computer program according to claim 16, wherein the language representation includes domain knowledge related to the handling of the object.

Images