Control device

Abstract

To provide a library presentation device, a library presentation method and a robot control system that present a library suitable for a target object.SOLUTION: A library presentation device (10) includes: an input information obtaining unit (131) that obtains input information including a target object; a result obtaining unit (132) that can obtain a result of a predetermined task with respect to the target object, using a plurality of libraries that can execute tasks with respect to the target object; an evaluation obtaining unit (133) that obtains an evaluation of the result of the application of the predetermined task; and a presentation-content determining unit (134) that determines out of the result of the application of the predetermined task using a second library that is different from a first library, of the plurality of libraries, on the basis of the evaluation of the result of the application of the predetermined task using the first library of the plurality of libraries.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2021-129342 (filed August 5, 2021), the entire disclosure of which is incorporated herein by reference. [Technical field]

[0002] The present disclosure relates to a library presentation device, a library presentation method, and a robot control system. [Background technology]

[0003] Conventionally, there are known devices that identify objects contained in images by learning using training images, etc. For example, Patent Document 1 discloses a robot that can be used by even unskilled workers to recognize various objects from images and perform tasks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-30407 Summary of the Invention

[0005] A library presentation device according to an embodiment of the present disclosure, an input information acquisition unit that acquires input information including an object; a result acquisition unit capable of acquiring a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; an evaluation acquisition unit that acquires an evaluation of a result of application of the predetermined task; and a presentation content determination unit that determines, based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries, an output of a result of application of the specified task using a second library among the plurality of libraries that is different from the first library.

[0006] A library display method according to an embodiment of the present disclosure includes: A library presentation method executed by a library presentation device, comprising: Obtaining input information including an object; obtaining a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; Obtaining an evaluation of a result of applying the predetermined task; The method includes determining an output of a result of application of the specified task using a second library different from the first library among the plurality of libraries based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries.

[0007] A robot control system according to an embodiment of the present disclosure includes: Robots and a robot control device that downloads a library presented by a library presentation device in order to control the robot; The library presentation device includes: an input information acquisition unit that acquires input information including an object; a result acquisition unit capable of acquiring a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; an evaluation acquisition unit that acquires an evaluation of a result of application of the predetermined task; and a presentation content determination unit that determines, based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries, an output of a result of application of the specified task using a second library among the plurality of libraries that is different from the first library. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a library presentation system including a library presentation device according to an embodiment. [Diagram 2]FIG. 2 is a schematic diagram showing a configuration example of a robot control system. [Diagram 3] FIG. 3 is a diagram showing an example of the configuration of a library group. [Figure 4] FIG. 4 is a sequence diagram illustrating an example of a processing procedure in the library presentation system. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a process for determining presentation contents of the library presentation device. [Figure 6] FIG. 6 is a sequence diagram illustrating another example of the processing procedure in the library presentation system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a library presentation device, a library presentation method, and a robot control system according to an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0010] (Library Presentation System Overview) FIG. 1 shows a configuration example of a library presentation system 1 including a library presentation device 10 according to the present embodiment. The library presentation device 10 is a device that presents a library suitable for a predetermined task when a user executes a predetermined task. The predetermined task is a process that is performed based on the result of recognizing an object, such as object recognition, identification, or display of the identification result, or recognition of the gripping position of an object. The library is a recognition model or a recognition program that can be used by a user. The user can use the library by downloading it via a network 40, for example. In the present embodiment, the library is configured as at least one machine learning model (trained model) that is generated by machine learning so that the above task can be executed, but is not limited thereto.

[0011] The library presentation device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an input information acquisition unit 131, a result acquisition unit 132, an evaluation acquisition unit 133, and a presentation content determination unit 134. The library presentation device 10 may be configured as a hardware computer such as a server. The components of the library presentation device 10 will be described in detail later.

[0012] The library presentation device 10 may configure a library presentation system 1 together with a library creation management device 20 and a communication terminal device 30 connected via a network 40. The network 40 is, for example, the Internet, but may also be a LAN (Local Area Network). For example, the library presentation device 10 and the library creation management device 20 may communicate via the LAN, and the library presentation device 10 and the communication terminal device 30 may communicate via the Internet.

[0013] The library generation management device 20 generates a library and manages a library group consisting of a plurality of libraries. The library generation management device 20 can generate a library by generating at least one trained model using learning data. The library generation management device 20 may generate a new library (e.g., a custom library described later) by machine learning. The library generation management device 20 may manage the library group by updating a library (additional learning or re-learning) using new learning data, and adding and deleting a library. In addition, the library generation management device 20 can execute a predetermined task using a specified library and output the execution result.

[0014] In this embodiment, the library generation management device 20 is capable of accessing the storage unit 12 of the library presentation device 10 and manages a group of libraries stored in the storage unit 12. In this embodiment, the library generation management device 20 is a computer separate from the library presentation device 10 and capable of communicating with the library presentation device 10, but is not limited to such a configuration. For example, the library generation management device 20 may be realized by a computer integrated with the library presentation device 10. In this case, the library generation management device 20 may have the components of the library presentation device 10, and the library presentation device 10 may have the components of the library generation management device 20.

[0015] The communication terminal device 30 is a general-purpose terminal device capable of communication, such as, for example, a smartphone or a tablet terminal, but is not limited thereto. As another example, the communication terminal device 30 may be configured as a part of a robot control system 100 (see FIG. 2) described later. The communication terminal device 30 generates input information (for example, an image of an object to be identified) and outputs the input information to the library presentation device 10. The communication terminal device 30 also presents to the user the presentation content (for example, a library suitable for a specific task) output by the library presentation device 10. The communication terminal device 30 includes a communication unit 31, a storage unit 32, a control unit 33, an input information generation unit 34, and a user interface unit 35. The components of the communication terminal device 30 will be described in detail later.

[0016] 2 is a schematic diagram showing an example of the configuration of a robot control system 100. The robot control system 100 includes a robot 2 and a robot controller 110. The robot 2 moves a work target 8 from a work start point 6 to a work target point 7 by an end effector 2B (gripping hand) at the end of an arm 2A. In other words, the robot controller 110 controls the robot 2 so that the work target 8 moves from the work start point 6 to the work target point 7. The robot control system 100 includes a camera 4. The camera 4 captures images of items and the like within an influence range 5 that may affect the operation of the robot 2.

[0017] The robot control device 110 recognizes the work object 8 present in the space where the robot 2 performs the work, based on the image captured by the camera 4. The recognition of the work object 8 is performed using a library (a trained model generated to recognize the work object 8) included in the above-mentioned library group. Before recognizing the work object 8, the robot control device 110 downloads the library via the network 40 and deploys it.

[0018] Here, it is preferable that the library group includes many libraries so that many types of work objects 8 can be recognized in the robot control system 100, for example. However, when there are many libraries, it may be difficult for a user to select a library suitable for the type of work object 8. Furthermore, before introducing the robot control system 100, a user may want to know whether an object can be appropriately recognized using a library in the library group. Furthermore, even after introducing the robot control system 100, a user may want to know whether a new work object 8 can be recognized.

[0019] As described below, the library presentation device 10 can output a library suitable for a predetermined task (for example, identifying an object and displaying the identification result) using input information (for example, an image including an object to be identified) from the communication terminal device 30. In this embodiment, the communication terminal device 30 is a tablet terminal carried by the user, but when the user has introduced the robot control system 100, it is possible to make part of the robot control system 100 (the robot control device 110, the camera 4, etc.) function as the communication terminal device 30.

[0020] (Configuration of the library presentation device) The components of the library presentation device 10 will be described in detail below. The communication unit 11 includes one or more communication modules connected to the network 40. The communication unit 11 may include a communication module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication unit 11 may include a communication module compatible with a wired LAN standard (1000BASE-T as an example). The communication unit 11 may include a communication module compatible with a wireless LAN standard (IEEE802.11 as an example).

[0021] The storage unit 12 is one or more memories. The memory may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these, and may be any memory. The storage unit 12 is, for example, built into the library presentation device 10, but may also be configured to be accessed from outside the library presentation device 10 via any interface.

[0022] The storage unit 12 stores various data used in various calculations performed by the control unit 13. The storage unit 12 may also store results and intermediate data of various calculations performed by the control unit 13.

[0023] As described above, the storage unit 12 is accessed by the library generation management device 20. That is, the storage unit 12 is shared by the library presentation device 10 and the library generation management device 20. In this embodiment, the storage unit 12 stores a library group (a plurality of libraries).

[0024] The control unit 13 is one or more processors. The processor may be, for example, a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these and may be any processor. The control unit 13 controls the overall operation of the library presentation device 10.

[0025] Here, the library presentation device 10 may have the following software configuration. One or more programs used to control the operation of the library presentation device 10 are stored in the storage unit 12. When the program stored in the storage unit 12 is read by the processor of the control unit 13, it causes the control unit 13 to function as an input information acquisition unit 131, a result acquisition unit 132, an evaluation acquisition unit 133, and a presentation content determination unit 134.

[0026] The input information acquisition unit 131 acquires input information from the communication terminal device 30. As described above, for example, when a user wants to know whether a task for a new object can be executed using a library in the library group, the user uses the library presentation device 10. The input information is information including the object. Specifically, in this embodiment, the input information is an image showing the object, but is not limited to an image. The input information may be, for example, information about the user or information about the robot 2 to be used, acquired from the communication terminal device 30 or the robot control device 110.

[0027] The result acquisition unit 132 acquires the result of applying a predetermined task to an object included in the input information. More specifically, the result acquisition unit 132 acquires the result of the library generation management device 20 executing a predetermined task using a specified library. In this embodiment, the predetermined task is recognition of an object and display of the recognition result. For example, when a predetermined task is executed for an image including multiple objects (such as industrial parts), recognition (identification of bolts, nuts, springs, etc.) is performed for each of the multiple objects, and the recognition result (names of bolts, nuts, springs, etc.) is overlaid on the image and output.

[0028] In addition, the result of applying a predetermined task using a specified library is not limited to an actual execution result, but may be a calculation result in a simulation. The simulation may include, for example, not only recognition of an object, but also grasping of an object that is difficult to actually execute. For example, the result acquisition unit 132 may recognize an object from input information, and then perform a 3D simulation according to the object to acquire the result of grasping. In addition, the 3D model in the 3D simulation may be synthesized from, for example, CAD data, RGB information, or depth information of the object.

[0029] The evaluation acquisition unit 133 acquires an evaluation for a result of applying a predetermined task. In this embodiment, the evaluation is at least one of precision and recall, but is not limited to these. The evaluation may include a judgment of "pass" or "fail" by the user's visual inspection. The evaluation acquisition unit 133 may acquire an evaluation input by the user to the communication terminal device 30. As another example, the evaluation acquisition unit 133 may acquire an expected result (at least one of expected precision and recall) input by the user to the communication terminal device 30 instead of the evaluation itself. When the evaluation acquisition unit 133 acquires an expected result for a predetermined task, the evaluation acquisition unit 133 may determine whether the result of applying the predetermined task is equal to or greater than the expected result based on the expected result, and may use the judgment as an evaluation. Note that, for example, in a case where the expected result is set in advance, the evaluation acquisition unit 133 may determine the applied result and use the result as an evaluation without receiving an input from the user.

[0030] The presentation content determination unit 134 determines the presentation content based on the evaluation acquired by the evaluation acquisition unit 133, and transmits the determined presentation content to the communication terminal device 30. When the evaluation is below a predetermined standard, the presentation content determination unit 134 may, for example, specify a different library or a combination of multiple libraries including the different library, apply a predetermined task, and set the different library or the combination of multiple libraries including the different library and the application result to the different library or the combination of multiple libraries including the different library as the presentation content. Note that, for example, the presentation content may include a library if the evaluation is below the predetermined standard. Details of the determination of the presentation content by the presentation content determination unit 134 will be described later.

[0031] (Configuration of communication terminal device) The components of the communication terminal device 30 will be described in detail below. The communication unit 31 includes one or more communication modules that connect to the network 40. The communication unit 31 may include a communication module that supports mobile communication standards such as 4G and 5G. The communication unit 31 may also include a communication module that supports a LAN standard. The communication unit 31 may include the same communication module as the communication unit 11, or may include a different communication module.

[0032] The storage unit 32 is one or more memories. The memory is, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these and may be any memory. The storage unit 32 stores various data used in various calculations performed by the control unit 33. The storage unit 32 may also store results and intermediate data of various calculations performed by the control unit 33. For example, the storage unit 32 may temporarily store input information generated by the input information generation unit 34 and transmitted (uploaded) to the library presentation device 10. For example, the storage unit 32 may temporarily store presentation content acquired from the library presentation device 10 via the communication unit 31.

[0033] The control unit 33 is one or more processors. The processor may be, for example, a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these and may be any processor. The control unit 33 controls the overall operation of the communication terminal device 30.

[0034] The input information generating unit 34 generates input information. As described above, in this embodiment, the input information is an image including an object to be identified. The input information generating unit 34 may be configured to include an imaging unit such as a camera. In this embodiment, the input information generating unit 34 is a camera. For example, when a user wants to know whether a new object can be recognized by the library presentation device 10, the user may capture an image including the new object with the camera of the communication terminal device 30 to generate input information.

[0035] The user interface unit 35 inputs information from the user and outputs information to the user. The user interface unit 35 includes, for example, a touch sensor. The touch sensor detects contact with the user's finger or a stylus pen, etc., and identifies the contact position. The touch sensor may be integrated with a display to configure a touch panel display. In this embodiment, the user interface unit 35 is a touch panel display. The content presented by the library presentation device 10 is displayed on the touch panel display. The user interface unit 35 may display a chat screen or the like to prompt input when the user decides on a specific task, decides on a library, or inputs an evaluation and expected results, etc.

[0036] (Libraries) In this embodiment, the library group stored in the memory unit 12 is roughly divided into three groups as shown in FIG. 3. The library group includes a general-purpose library group, a specialized library group, and a custom library group. The general-purpose library group includes a plurality of general-purpose libraries for recognizing objects in a large classification (first classification). The specialized library group includes a plurality of specialized libraries for recognizing objects in a smaller classification (second classification). The second classification is a subdivision of one item of the first classification. In addition, the custom library group includes a plurality of custom libraries generated by learning data prepared by the user.

[0037] In the example of FIG. 3, the first category is industrial parts, stationery, etc. The second category is bolts, nuts, pens, scissors, etc. For example, the general-purpose library "industrial parts" is a library for recognizing bolts, nuts, springs, etc. For example, the specialized library "bolt" is a library for recognizing bolt types such as bolt A, bolt B, and bolt C. The specialized libraries "bolt" and "nut" are associated with the general-purpose library "industrial parts". Such association between the general-purpose library and the specialized library is stored in the storage unit 12 as association information. That is, the storage unit 12 stores association information that defines the relationship between multiple libraries. For the custom library group, the association between the custom library and the user, the association between the general-purpose library and the custom library, or the association between the specialized library and the custom library may be stored in the storage unit 12 as association information. Furthermore, similarities between multiple general-purpose libraries, similarities between multiple specialized libraries, etc. may be stored in the storage unit 12 as association information. Furthermore, the storage unit 12 may store, as related information, a usage history of a general-purpose library, a specialized library, or a custom library, or a combination of multiple general-purpose libraries, multiple specialized libraries, or multiple custom libraries.

[0038] The presentation content determination unit 134 may cause the result acquisition unit 132 to acquire the application result of a predetermined task using, for example, a second library having a predetermined relationship with a library (first library) designated by the user based on the above-mentioned related information. For example, the user may use the library presentation device 10 to confirm whether a "bolt C'", which is an improved product of "bolt C", can be identified using a library in the library group. In this case, the user may specify "industrial parts" in the general-purpose library because the user does not know the details of the type of library. In such a case, the presentation content determination unit 134 may cause the library generation management device 20 to not only recognize "bolt C'" using "industrial parts" (general-purpose library, first library), but also recognize "bolt C'" using "bolt" (specialized library, second library) based on the related information. Then, the presentation content determination unit 134 may cause the result acquisition unit 132 to acquire each execution result. For example, the presentation content determination unit 134 may cause the execution result of a predetermined task using the second library to be presented when the evaluation of the execution result of a predetermined task using the first library is below a predetermined standard. The case where the evaluation is below the predetermined standard means, for example, that the precision rate and recall rate are below the expected result. By using the related information, the presentation content determination unit 134 can select a candidate library suitable for the execution of a predetermined task even if the user does not know much about the types of libraries. Here, for example, in the case where there are multiple second libraries related to the first library, the presentation content determination unit 134 may execute and evaluate each of the multiple second libraries. In addition, the presentation content determination unit 134 may cause the result acquisition unit 132 to acquire the application result of the predetermined task using the subsequent library (e.g., the second library) when the evaluation of the execution result of the predetermined task using the previous library (e.g., the first library) is below the predetermined standard.

[0039] Furthermore, the presentation content determination unit 134 may cause the result acquisition unit 132 to acquire the execution result of a predetermined task using a part of the second libraries based on, for example, the relevance of the related information. The relevance may be, for example, the frequency of use of a library used in the past by the user or the robot 2 to be used. Furthermore, when a plurality of second libraries are related to the first library, the presentation content determination unit 134 may set priorities to the plurality of second libraries and cause the result acquisition unit 132 to acquire the execution result of a predetermined task using a second library with a high priority. Furthermore, in this case, the presentation content determination unit 134 may sequentially evaluate the second libraries, and when an evaluation equal to or higher than a predetermined standard is obtained, the result acquisition unit 132 may complete the acquisition of the execution result. The priority may be set based on, for example, the frequency of use of a library used in the past by the user or the robot 2 to be used.

[0040] Furthermore, the presentation content determination unit 134 may cause the result acquisition unit 132 to acquire the execution result of a predetermined task using a custom library (third library) previously generated by the user based on the related information. In addition, when there are multiple third libraries related to the first library, the presentation content determination unit 134 may execute and evaluate each of the multiple third libraries. In addition, the presentation content determination unit 134 may cause, for example, the third library to be presented. In addition, the first library may be used in combination with another library (e.g., the second library). In addition, the second library may be used in combination with another library (e.g., the third library). In other words, the result acquisition unit 132 may acquire the execution result of a predetermined task when multiple libraries are combined. For example, the presentation content determination unit 134 may compare the evaluations when multiple libraries are combined, and cause the best combination of libraries and the execution result of a predetermined task using them to be presented. For example, the presentation content determination unit 134 may cause the execution result of a predetermined task when the third library and the second library are combined to be presented. For example, the "industrial parts" in the first library and the "bolts" in the second library may be combined (both used) to execute object recognition as a given task. In this example, if the object is "bolt B", the execution result is expected to be correctly recognized as "bolt B", but if the object is "nut B", it may be recognized simply as a nut.

[0041] The presentation content determination unit 134 may cause a proposal for generating a new library to be presented when an evaluation of the execution result of a predetermined task using the second library is below a predetermined standard. The new library is a new custom library. For example, the user can create learning data from an image including "bolt C'" and cause the library generation management device 20 to generate a new library using the learning data.

[0042] In this way, the presentation content determination unit 134 not only causes the execution result of a specified task using a library (first library) specified by the user to be presented, but also causes the second library, the third library, and a combination of these libraries to be presented along with the execution result depending on the evaluation. Furthermore, the presentation content determination unit 134 causes a proposal for the creation of a new library to be presented. The proposal for the creation of a new library may be made when the "Bolt C'" in the above example is not recognized, when "Bolt C'" is misclassified as "Bolt A" or the like, and when "Bolt C'" is not classified in detail by type.

[0043] Here, in the above example, the first library is a general-purpose library, the second library is a specialized library, and the third library is a custom library, but these correspondences are not limited. For example, the second library may be a general-purpose library, and the third library may be a specialized library. In this case, for example, if the first library specified by the user is "industrial products", "stationery" may be applied as the second library, and "pen" may be applied as the third library. Also, for example, the first library may be a general-purpose library, and the second library may be a custom library. In this case, for example, if the first library specified by the user is "industrial products", "bolt C'" may be applied as the second library. Also, the first library may be a specialized library or a custom library. In this case, for example, if the first library specified by the user is "bolt A", "bolt B" may be applied as the second library, and "bolt C'" may be applied as the third library, or if the first library specified by the user is "bolt C'", "bolt C" may be applied as the second library, and "bolt D'" may be applied as the third library.

[0044] In the above example, the third library is set based on the association information with the first library, but the third library may be set based on the association information with the second library, or based on the association information of both the first library and the second library. In this case, for example, if the first library is "industrial products", the second library may be "stationery" and the third library may be "pens", or if the first library is "industrial products" and the second library is "bolt C", the third library may be "bolt C'".

[0045] In addition, the combination of libraries is not particularly limited. In the above example, the result acquisition unit 132 acquires the result of executing a predetermined task when, for example, multiple general-purpose libraries or a general-purpose library and a specialized library are combined. The result acquisition unit 132 may acquire the result of executing a predetermined task when, for example, multiple specialized libraries, multiple custom libraries, a specialized library and a custom library, or a general-purpose library and a custom library are combined.

[0046] (Library Presentation Method) Fig. 4 is a sequence diagram showing an example of a processing procedure in the library presentation system 1. Fig. 5 is a flowchart showing an example of a process for determining the presentation contents of the library presentation device 10. The process of the library presentation method executed by the library presentation device 10 according to this embodiment will be described with reference to Figs. 4 and 5.

[0047] A user using the library presentation device 10 determines a predetermined task and inputs it to the communication terminal device 30 (step S1). Here, the predetermined task will be described as being the recognition of objects in an image including "Bolt C'" and other industrial parts and the display of the recognition results, as in the above example.

[0048] The user uses the camera of the communication terminal device 30 to capture images of the "bolt C'" and other industrial parts, and generates captured images that are input information (step S2).

[0049] The communication terminal device 30 transmits the input information (captured image) to the library presentation device 10 (step S3). The input information acquisition unit 131 of the library presentation device 10 acquires the input information.

[0050] The user selects a library and inputs it into the communication terminal device 30 (step S4). Here, since the user does not know the details of the library type, it is assumed that the user has selected "industrial parts" from the general-purpose library.

[0051] The communication terminal device 30 transmits the user-selected library to the library presentation device 10 (step S5). That is, the communication terminal device 30 transmits to the library presentation device 10 information that the library selected by the user is “industrial parts” of the general-purpose library.

[0052] The presentation content determination unit 134 of the library presentation device 10 transmits the input information obtained via the input information acquisition unit 131 and information on the specified library ("industrial parts" in the general-purpose library) to the library generation management device 20, and causes it to execute a specified task (step S6).

[0053] The library generation management device 20 executes a predetermined task (step S7).

[0054] The library generation management device 20 transmits the execution result of the predetermined task to the library presentation device 10 (step S8). The result acquisition unit 132 of the library presentation device 10 acquires the execution result of the predetermined task.

[0055] The presentation content determination unit 134 of the library presentation device 10 transmits the execution result of the predetermined task to the communication terminal device 30 (step S9). The execution result of the predetermined task is displayed on the user interface unit 35 (touch panel display) of the communication terminal device 30.

[0056] The user evaluates the execution result of a given task and inputs the evaluation to the communication terminal device 30 (step S10). The evaluation may be a precision rate and a recall rate. The user may further input an expected result (such as an expected precision rate and a recall rate) to the communication terminal device 30 as the evaluation.

[0057] The communication terminal device 30 transmits the evaluation input by the user to the library presentation device 10 (step S11). The evaluation acquisition unit 133 of the library presentation device 10 acquires the evaluation. Here, if the expected result is set in advance, the evaluation acquisition unit 133 may proceed to the process of step S12 without receiving an evaluation from the user.

[0058] The presentation content determination unit 134 of the library presentation device 10 executes a process of determining the presentation content based on the evaluation obtained via the evaluation acquisition unit 133 (step S12). The presentation content determination process is executed according to the flowchart of FIG. 5, as described later.

[0059] The presentation content determination unit 134 of the library presentation device 10 transmits the presentation content to the communication terminal device 30 and displays it on the user interface unit 35 (touch panel display) (step S13). Based on the presentation content, the user can check whether the object can be identified using a library in the library group.

[0060] The library presentation device 10 executes the presentation content determination process, for example, as follows. When the evaluation by the evaluation acquisition unit 133 is less than a predetermined standard (Yes in step S21), the presentation content determination unit 134 instructs the library generation management device 20 to execute a predetermined task using a library other than the user-selected library (step S22). The presentation content determination unit 134 instructs the library generation management device 20 to execute a predetermined task using a library related to the library selected by the user. As described above, the presentation content determination unit 134 selects a library based on the related information stored in the storage unit 12. For example, when the library selected by the user is "industrial parts" of the general-purpose library, "bolts" of the specialized library and a custom library of the same user are selected. In addition, the presentation content determination unit 134 may instruct to combine multiple libraries to execute a predetermined task. Here, the predetermined standard may be determined in advance, or may be determined based on the expected result input in step S10, for example.

[0061] The result acquisition unit 132 acquires the execution result of a predetermined task using a library other than the user-selected library (hereinafter also referred to as a "non-user-selected library") (step S23).

[0062] The presentation content determination unit 134 sets the execution result of the predetermined task using the non-user selected library as the presentation content (step S24). The non-user selected library includes, for example, a specialized library that the user does not know. The specialized library is more likely to satisfy the user's request because it recognizes objects in smaller classifications than a general-purpose library. In this way, the presentation content determination unit 134 can present a more suitable library even if the evaluation of the predetermined task using the user selected library is low. Here, the presentation content determination unit 134 may evaluate the execution result of the predetermined task using the non-user selected library based on the expected result (such as expected precision and recall) obtained via the evaluation acquisition unit 133. In this case, the presentation content determination unit 134 may set the execution result of the predetermined task using the non-user selected library as the presentation content after evaluating it.

[0063] Furthermore, if the evaluation is equal to or greater than a predetermined standard (No in step S21), the presented content determination unit 134 sets the fact that the user-selected library is linked to a predetermined task as the presented content (step S25). For example, the predetermined standard may be a standard for the accuracy rate, such as a matching rate of 90% or more. In this example, if the evaluation is equal to or greater than the predetermined standard, the presented content determination unit 134 presents the user-selected library, since the accuracy rate of the user-selected library is high and there are few misclassifications. Furthermore, if the evaluation is less than the predetermined standard, the presented content determination unit 134 presents a more suitable library (a library not selected by the user), since the accuracy rate of the user-selected library is low.

[0064] Here, in the example of Fig. 4, the library presentation device 10 waits for the user's evaluation before executing a predetermined task using a library not selected by the user, but it is also possible to execute the task without waiting for the user's evaluation. Fig. 6 is a sequence diagram showing another example of the processing procedure in the library presentation system 1.

[0065] Steps S101 to S105 in FIG. 6 are the same as steps S1 to S5 in FIG. 4, and therefore the description thereof will be omitted.

[0066] The user inputs the expected results to the communication terminal device 30 (step S106). The expected results are, for example, the expected precision rate and recall rate, as described above.

[0067] The communication terminal device 30 transmits the expected result input by the user to the library presentation device 10 (step S107). The evaluation acquisition unit 133 of the library presentation device 10 acquires the expected result.

[0068] The presentation content determination unit 134 of the library presentation device 10 issues an instruction to execute a predetermined task that expands the library (step S108). The expansion of the library means designating the use of a non-user selected library in addition to the user selected library. The non-user selected library may be selected based on related information.

[0069] The library creation management device 20 executes a predetermined task (step S109). The library creation management device 20 executes a plurality of predetermined tasks using a plurality of libraries.

[0070] The library generation management device 20 transmits a plurality of execution results of a predetermined task to the library presentation device 10 (step S110). The result acquisition unit 132 of the library presentation device 10 acquires the plurality of execution results.

[0071] The evaluation acquisition unit 133 of the library presentation device 10 does not execute the reception process when a predetermined task using a non-user selected library is executed without waiting for a user evaluation. For example, the evaluation acquisition unit 133 determines whether each execution result is equal to or exceeds the expected result based on a preset expected result and a plurality of execution results, and acquires an evaluation based on the determination.

[0072] The presentation content determination unit 134 determines the presentation content based on the evaluation of the multiple execution results (determination of whether the result is above or equal to the expected result) (step S111). For example, the presentation content determination unit 134 may compare the evaluations of the multiple execution results and set the most highly evaluated library and the execution result of a predetermined task using the library as the presentation content.

[0073] The presentation content determination unit 134 of the library presentation device 10 transmits the presentation content to the communication terminal device 30, and causes it to be displayed on the user interface unit 35 (touch panel display) (step S112).

[0074] As described above, the library presentation device 10 and the library presentation method according to the present embodiment can present a library suitable for an object. Even before the introduction of the robot control system 100 that deploys and operates the library, or even when recognition of a new object is required, the user can know whether a library suitable for the library group is available.

[0075] Although the embodiments of the library presentation device 10, the library presentation method, and the robot control system have been described above, the present disclosure may also be embodied as a system, a program, or a storage medium on which a program is recorded. Examples of the storage medium include an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card.

[0076] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be in the form of a program module incorporated into an operating system, etc. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.

[0077] The drawings for explaining the embodiments according to the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0078] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.

[0079] The constituent elements and process steps described in this disclosure may be combined in any combination except those in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.

[0080] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.

[0081] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In the configurations distinguished by descriptions such as "first" and "second" in this disclosure, the numbers in the configurations can be exchanged. For example, the first classification can exchange the identifiers "first" and "second" with the second classification. The identifiers are exchanged simultaneously. The configurations are still distinguished after the identifiers are exchanged. The identifiers may be deleted. The configurations from which the identifiers have been deleted are distinguished by the symbols. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers. For example, the scope of the present invention also includes those in which the identifiers "first" and "second" are exchanged in the claims, such as the first library and the second library. [Explanation of symbols]

[0082] 1 Library Presentation System 2. Robot 2A Arm 2B End effector 4. Camera 5. Scope of impact 6. Starting point 7. Work target point 8 Work Object 10 Library presentation device 11 Communications Department 12 Storage section 13 Control section 20 Library creation management device 30 Communication terminal equipment 31 Communications Department 32 Storage section 33 Control Unit 34 Input information generation unit 35 User Interface Section 40 Network 100 Robot Control System 110 Robot control device 131 Input information acquisition unit 132 Result acquisition part 133 Evaluation Department 134 Presentation content determination section

Claims

1. an input information acquisition unit that acquires input information including an object; a result acquisition unit capable of acquiring a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; an evaluation acquisition unit that acquires an evaluation for a result of applying the predetermined task; a presentation content determination unit that determines, based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries, an output of a result of application of the specified task using a second library among the plurality of libraries that is different from the first library.

2. the result acquisition unit acquires a result of applying the predetermined task when the plurality of libraries are combined, The library presentation device according to claim 1 , wherein the presentation content determination unit presents a result of application of the predetermined task using a combination of the second library and a third library different from the first library and the second library.

3. The library presentation device according to claim 1 , wherein the evaluation acquisition unit, when acquiring an expected result for the predetermined task, acquires an evaluation for a result of applying the predetermined task based on the expected result.

4. The library presentation device according to claim 1 or 2, wherein the presentation content determination unit causes the result acquisition unit to acquire the results of application of the specified task using the second library having a specified relationship with the first library based on association information defining the relationship between the libraries.

5. The library presentation device according to claim 1 or 2, wherein the plurality of libraries include a general-purpose library for recognizing the object in a first classification, and a specialized library for recognizing the object in a second classification that subdivides one item of the first classification.

6. The library presentation device according to claim 5 , wherein the result acquisition unit acquires a result of application of the predetermined task when a plurality of the general-purpose libraries or a combination of the general-purpose library and the specialized library are combined.

7. The library presentation device according to claim 1 , wherein the presentation content determination unit is configured to present a proposal for creating a new library when an evaluation of a result of application of the predetermined task using the second library is below a predetermined standard.

8. The library, the first library, and the second library are configured as a machine learning model that recognizes the object; The library presentation device according to claim 1 , wherein the predetermined task is performed based on a result of recognizing the object.

9. A library presentation method executed by a library presentation device, comprising: Obtaining input information including an object; obtaining a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; Obtaining an evaluation of a result of applying the predetermined task; A library presentation method comprising: determining an output of a result of application of the specified task using a second library different from the first library among the plurality of libraries based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries.

10. Robots and a robot control device that downloads a library presented by a library presentation device in order to control the robot; The library presentation device includes: an input information acquisition unit that acquires input information including an object; a result acquisition unit capable of acquiring a result of applying a predetermined task to the object using a plurality of libraries capable of executing the task on the object; an evaluation acquisition unit that acquires an evaluation for a result of applying the predetermined task; a presentation content determination unit that determines, based on an evaluation of a result of application of the specified task using a first library among the plurality of libraries, an output of a result of application of the specified task using a second library among the plurality of libraries that is different from the first library.

Images