Information processing apparatus, information processing method, and information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program capable of individually coping with a problem relating to recognition accuracy or robustness of a learned model when the problem occurs.SOLUTION: In an information processing system 100 including an information processing device 10, a picking control device 20, a picking device 30, and a workpiece (object) 40, the information processing device inputs data to each of a plurality of learned models, calculates a probability of a predetermined event based on an output result of each of the plurality of learned models with respect to the input data, and recognizes the event based on the calculation result. The picking control device uses the acquired detection result to generate a control signal related to a picking plan of the workpiece to be picked, and outputs the control signal to the picking device. The picking apparatus the target workpiece and moves the workpiece to the target container in accordance with the acquired control signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, technological developments related to machine learning models have been actively carried out (for example, Patent Documents 1 to 5). Patent Documents 1 to 5 describe methods for improving the recognition accuracy and robustness of models. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 286847 [Patent Document 2] Patent Publication No. 2021-056803 [Patent Document 3] Japanese Patent Publication No. 2023-028431 [Patent Document 4] Japanese Patent Publication No. 2023-058276 [Patent Document 5] Japanese Patent Application Publication No. 2023-168240 Summary of the Invention [Problem to be solved by the invention]

[0004] Improving the recognition accuracy of a model is obviously important, but no model can guarantee 100% accuracy, and problems are inevitable. Furthermore, as is known from the no-free-lunch theorem, there is no optimal solution for all problems. Therefore, simply improving the recognition accuracy and robustness of a single model, as in the conventional technology described above, makes it difficult to use that model when a problem occurs, and the model must be rebuilt. However, rebuilding a model and verifying the operation of the rebuilt model takes time. Therefore, relying on a single model makes it difficult to provide temporary solutions, such as replacement, compensation, or substitution, when a problem occurs.

[0005] An object of one aspect of the present disclosure is to provide an information processing device that can individually address problems that occur. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure is an information processing device that recognizes a predetermined event by inputting data into a plurality of trained models, and is equipped with at least one processor, wherein the processor inputs the data to each of the plurality of trained models, calculates the likelihood of the event based on the output results of each of the plurality of trained models for the input data, and recognizes the event based on the calculation results.

[0007] In order to solve the above problem, an information processing method according to one embodiment of the present disclosure is an information processing method executed by one or more information processing devices, and includes an input step of inputting data to each of a plurality of trained models, a calculation step of calculating the likelihood of the event based on the output results of each of the plurality of trained models for the input data, and a recognition step of recognizing the event based on the calculation results.

[0008] In order to solve the above problem, an information processing program according to one embodiment of the present disclosure causes a computer to execute an input process for inputting data into each of a plurality of trained models, a calculation process for calculating the likelihood of a predetermined event based on the output results of each of the plurality of trained models for the data, and a recognition process for recognizing the event based on the calculation results.

[0009] The information processing device according to each aspect of the present disclosure may be realized by a computer. In this case, the information processing program that causes the computer to operate as each part (software element) of the information processing device to realize the information processing device on the computer, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present disclosure. [Effects of the Invention]

[0010] According to one aspect of the present disclosure, it is possible to deal with each problem individually depending on the problem that has occurred. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an overview of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 is a block diagram showing an example of functions of an information processing device. [Figure 4] FIG. 10 is a block diagram illustrating an example of integrating the outputs of each trained model. [Figure 5] FIG. 10 is a block diagram showing an example of a method for generating and operating each trained model. [Figure 6] FIG. 10 is a diagram illustrating an example of an integration result. [Figure 7] FIG. 10 is a diagram illustrating an example of an integration result. [Figure 8] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 9]10A and 10B are diagrams illustrating an example of detection when image data and point cloud data are input. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, identical or substantially identical components are designated by the same reference numerals, and description thereof will be omitted.

[0013] (An example of an overview of the information processing system 100) Fig. 1 is a diagram illustrating an example of an overview of an information processing system 100. As shown in Fig. 1, the information processing system 100 includes an information processing device 10, a picking control device 20, a picking device 30, and a work 40. An example of a scenario in which such information processing system 100 is used is a scenario in which a large number of different items are efficiently picked into shipping containers according to their destinations in a logistics warehouse or the like.

[0014] The information processing device 10 acquires an image captured by a camera (not shown), performs predetermined processing on the image using multiple trained models, and detects objects from within the image. Details of the processing using multiple trained models will be described later. In this embodiment, "detecting an object" refers to detecting, for example, the "size of the object," the "position of the object," the "orientation of the object," the "type of object," and the "number of objects." Furthermore, the object refers to the item to be picked as described above, and an example of an item is a commodity. Such an item may be referred to as a "work" below. The information processing device 10 outputs the detection result to the picking control device 20.

[0015] The picking control device 20 generates a control signal related to a picking plan, such as which container and which position the workpiece 40 to be picked should be moved to, using the detection results acquired from the information processing device 10. The picking control device 20 outputs the generated control signal to the picking device 30.

[0016] The picking device 30 is typically a robot arm installed in a logistics warehouse, etc. The picking device 30 picks up the target workpiece 40 and moves it to the target container in accordance with a control signal acquired from the picking control device 20.

[0017] The information processing device 10, the picking control device 20, and the picking device 30 may be configured as an integrated device or as separate devices. If they are configured as separate devices, the picking device 30 needs to be installed in a logistics warehouse or the like, but the installation locations of the information processing device 10 and the picking control device 20 are not limited. For example, the information processing device 10 and the picking control device 20 may be installed in a management center of a business operator that provides picking services. In this case, it is sufficient that the information processing device 10, the picking control device 20, and the picking device 30 are configured to be able to communicate with each other via a predetermined wireless network (e.g., the Internet) or the like.

[0018] Note that, hereinafter, the information processing device 10 will be mainly described as a device that performs processing to detect an object (here, workpiece 40) from within an image, but the events that the information processing device 10 detects and recognizes are not limited to objects. For example, any event that can be sensed by any sensor, such as "animal cries," "failure sounds," "taste," "temperature," "humidity," or "seismic intensity," and whose sensed data can be processed by a computer, can be recognized by the information processing device 10.

[0019] (Hardware configuration of information processing device 10) Next, an example of the hardware configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the information processing device 10.

[0020] 2, the information processing device 10 includes a processor 11, a memory 12, a storage device 13, an input / output I / F 14, and a communication I / F 15. These components are connected via a bus 16 so as to be able to communicate with each other.

[0021] The memory 12 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and stores programs executed by the processor 11, information used by the programs, etc. The processor 11 has a CPU (Central Processing Unit) that performs arithmetic processing, reads programs from the ROM, and executes the programs using the RAM as a working area. Although one processor 11 is shown in FIG. 2, this is not limiting, and multiple processors 11 may be provided. Furthermore, the computer-readable recording medium is not limited to ROM and RAM, and may include an EPROM (Erasable Programmable ROM), an EEPROM (registered trademark), etc.

[0022] The storage device 13 is configured with a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like, and stores various programs and various data. FIG. 2 shows an example in which a first trained model 131, a second trained model 132, and a third trained model 133 are stored in the storage device 13. However, the first trained model 131, the second trained model 132, and the third trained model 133 do not necessarily have to be stored in the storage device 13. For example, the first trained model 131, the second trained model 132, and the third trained model 133 may be stored in a cloud server or the like. In this case, the information processing device 10 can access the cloud server or the like and use the first trained model 131, the second trained model 132, and the third trained model 133. The first trained model 131, the second trained model 132, and the third trained model 133 are models generated in advance by learning predetermined data. In this embodiment, an example using three trained models is described, but the number of trained models employed is not limited to three, and may be two, or four or more.

[0023] The input unit of the input / output I / F 14 is typically a keyboard or a mouse, but is not limited to these and may be any configuration that allows various data to be input. For example, the input unit may be a microphone, a touch panel, etc. The output unit of the input / output I / F 14 is typically a display, but is not limited to these and may be any configuration that allows various data to be output. For example, the output unit may be a speaker, a warning light, etc.

[0024] The communication I / F 15 is implemented as hardware such as a network adapter, various communication software, or a combination thereof, and is configured to be able to realize wireless communication via a predetermined wireless network.

[0025] (Functions of the processor 11 of the information processing device 10) Next, an example of the functions of the processor 11 of the information processing device 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functions of the processor 11 of the information processing device 10. The processor 11 of the information processing device 10 executes a program stored in the memory 12, thereby functioning as a data acquisition unit 111, a data input unit 112, a weighting unit 113, a score adjustment unit 114, an aggregation unit 115, an object detection unit 116, and an output unit 117. Note that what is described as "unit" here may be rephrased as "circuit", "step", "procedure", "processing", etc.

[0026] The data acquisition unit 111 acquires image data captured by a camera from above of a container in which a workpiece 40 to be picked is placed. As an example of a method for acquiring image data, the data acquisition unit 111 may acquire image data directly from a camera or may acquire the image data by accessing the storage device 13 in which the image data is stored. Alternatively, if the image data is stored on a cloud server or the like, the data acquisition unit 111 may acquire the image data by accessing the cloud server or the like. The data acquisition unit 111 outputs the acquired image data to the data input unit 112. Note that when the information processing device 10 recognizes other events, such as the above-mentioned "animal sounds," "fault sounds," "taste," "temperature," "humidity," and "seismic intensity," rather than object detection, these events are sensed by the following sensors: "Animal sounds" and "fault sounds" are sensed by a microphone. "Taste" is sensed by a taste sensor. "Temperature" is sensed by a temperature sensor. "Humidity" is sensed by a humidity sensor. "Seismic intensity" is sensed by a seismometer. The method for recognizing these events is realized by a process similar to that for object detection, which will be described below.

[0027] The data input unit 112 inputs the image data acquired from the data acquisition unit 111 into each of the first trained model 131, the second trained model 132, and the third trained model 133. This results in an output from each of the first trained model 131, the second trained model 132, and the third trained model 133 in response to the input. Note that the processing by the data input unit 112 corresponds to "input processing."

[0028] The weighting unit 113 weights the outputs of the first trained model 131, the second trained model 132, and the third trained model 133 based on mAP (Mean Average Precision). mAP is a type of performance evaluation index in machine learning tasks such as object detection and image recognition. As an example of a specific weighting method, assume that the output scores are 95% for the first trained model 131, 76% for the second trained model 132, and 19% for the third trained model 133. In this case, the weighting unit 113 multiplies the output score of the first trained model 131 by 0.5, the output score of the second trained model 132 by 0.4, and the output score of the third trained model 133 by 0.1. The weighting unit 113 outputs the weighted scores to the score adjustment unit 114.

[0029] The score adjustment unit 114 calculates overlapping bounding boxes using Soft-NMS (Non-Maximum Suppression) for the scores weighted by the weighting unit 113 and adjusts the scores. This is mainly performed to improve overlapping portions where the outputs of each trained model overlap. Soft-NMS is a method for processing overlapping bounding boxes in machine learning tasks such as object detection and image recognition. A bounding box (also called a BBOX) is a box that encloses an area in which an object is captured, with a rectangular or square shape, in machine learning tasks such as object detection and image recognition. Such bounding boxes represent the position (coordinates) and size of the object and are used to identify and track detected objects. The score adjustment unit 114 increases the scores of plausible bounding boxes to eliminate overlaps and decreases the scores of other bounding boxes. More specifically, the score adjustment unit 114 calculates the Intersection over Union (IoU) for the bounding box with the highest score and adjusts the score to eliminate overlaps. Alternatively, the score adjustment unit 114 may adjust the scores of the overlapping portions to remove the overlapping. IoU is an index for evaluating the degree of overlapping of bounding boxes in machine learning tasks such as object detection and image recognition.

[0030] The aggregation unit 115 aggregates the inference results of each trained model. This is done primarily to improve non-overlapping areas. The score adjustment method by the score adjustment unit 114 described above eliminates overlapping bounding boxes, but does not consider non-overlapping bounding box areas. This means that noise output by low-reliability trained models may remain. In other words, there is a possibility that low-reliability areas (noise) output by only one trained model may remain. Therefore, the aggregation unit 115 calculates the likelihood of the bounding boxes output by each trained model based on the IoU, score, coordinates, number of trained models, and reliability of the trained models. By retaining only outputs with high likelihood calculated in this way, the aggregation unit 115 can remove noise and improve non-overlapping bounding box areas.

[0031] The algorithm used by the aggregation unit 115 according to this embodiment is based on WBF (Weighted Box Fusion), but it adjusts weighting based on the reliability of each trained model based on mAP, not just the number of trained models. WBF is typically a method of removing bounding boxes that do not overlap among multiple trained models (bounding boxes that are output by only one trained model are treated as noise). The process by the score adjustment unit 114 reduces the score of overlapping parts, while the process by the aggregation unit 115 reduces the score of non-overlapping parts. Combining the processes by the score adjustment unit 114 and the aggregation unit 115 directly would significantly reduce all scores. However, in this embodiment, by taking into account the reliability (weighting) of the trained models, the process by the score adjustment unit 114 and the process by the aggregation unit 115 are integrated to remove overlapping parts and non-overlapping parts, thereby obtaining the expected output. As a result, this embodiment can effectively remove noise while also dealing with cases where only one trained model outputs, such as when an item is not detected by the first trained model 131 and the second trained model 132 but is detected by the third trained model 133, thereby minimizing undetection and leaving only plausible outputs. This reduces the risk of the robot arm abnormally stopping because a product is recognized as not being present in a container, even though the product is actually present.

[0032] The processes performed by the weighting unit 113, the score adjusting unit 114, and the aggregating unit 115 correspond to "calculation processes."

[0033] The object detection unit 116 integrates the results of processing performed by the score adjustment unit 114 and the aggregation unit 115 on the output results of each model, and detects the workpiece 40 to be picked. The processing by the object detection unit 116 corresponds to "recognition processing."

[0034] The output unit 117 outputs the detection result by the object detection unit 116 to the picking control device 20. In this way, the desired workpiece 40 is picked up.

[0035] (Example of integrating the outputs of each trained model) 4 is a block diagram showing an example of integrating the outputs of each trained model. Trained models 1 to 3 are set in a predetermined module list. Models 1 to 3 correspond to, for example, the first trained model 131, the second trained model 132, and the third trained model 133 described above.

[0036] The original data (e.g., image data) to be input to models 1 to 3 are prepared as subsets 1 to 3. The weighted output of each model based on mAP is processed using Soft-NMS and WBF, and the outputs of each model are integrated to remove bounding boxes with low scores and leave bounding boxes with high scores.

[0037] (Examples of machine learning model generation and operation) Figure 5 is a block diagram showing an example of a method for generating and operating the first trained model 131, the second trained model 132, and the third trained model 133. For ease of explanation, in Figure 5, the first trained model 131 will be referred to as the "main model 51," the second trained model 132 as the "large work model 54," and the third trained model 133 as the "small work model 57."

[0038] First, large-scale training data 50 is prepared. A main model 51 constructed by training the large-scale training data 50 for approximately three weeks is output to a test environment, and a test evaluation 52 is performed. Next, based on recognition errors and the like discovered in the test evaluation 52, small-scale training data 53 is prepared for a model to improve the recognition errors and the like. A large work model 54 constructed by re-training the small-scale training data 53 for approximately six hours is output to a test environment, and a test evaluation 55 is performed. Next, based on recognition errors and the like discovered in the test evaluation 55, small-scale training data 56 is prepared for a model to improve the recognition errors and the like. A small work model 57 constructed by re-training the small-scale training data 56 for approximately two hours is output to a test environment, and a test evaluation 58 is performed.

[0039] The large work model 54 is a model fine-tuned using small-scale training data 53, which is data selected from the large-scale training data 50 that has a low accuracy rate. The small-scale training model 57 is a model fine-tuned using small-scale training data 56, which is data selected from the small-scale training data 53 that has a low accuracy rate. The data volume is large-scale training data 50 > small-scale training data 53 > small-scale training data 56. By improving the training method in this way, by selecting only a small amount of data with a low accuracy rate and re-training it, the training time can be significantly reduced. Note that the large-scale training data 50, the small-scale training data 53, and the small-scale training data 56 may be supervised data or unsupervised data.

[0040] The main model 51, large work model 54, and small work model 57 generated through learning and relearning are deployed in an actual environment and put into operation.

[0041] (An example of the integrated results) Next, an example of the integration result will be described with reference to FIGS.

[0042] FIG. 6 shows output results 71, 72, and 73 when image data of a container 80 photographed from above by a camera is input to a first trained model 131, a second trained model 132, and a third trained model 133, and an ensemble result 74 obtained by processing these output results 71, 72, and 73 by the score adjustment unit 114 and aggregation unit 115 described above and integrating the results of the processing. Note that in FIG. 6, as described in FIG. 5, the first trained model 131 is treated as the "main model 51," the second trained model 132 as the "large work model 54," and the third trained model 133 as the "small work model 57." Also, in FIG. 6, if the score is equal to or greater than a preset threshold, the work is represented by line hatching. If the score is less than the threshold, the work is represented by dot hatching. If the score is equal to or greater than the threshold, it means that the work has been detected generally correctly, and if the score is less than the threshold, it means that the work has not been detected as expected.

[0043] As shown in output result 71, the first trained model 131 generally correctly detected all of the six works 41 to 46 in the container 80, except for works 43 and 46. Also, as shown in output result 72, the second trained model 132 generally correctly detected all of the six works 41 to 46 in the container 80. Also, as shown in output result 73, the third trained model 133 did not obtain the expected detection results for any of the six works 41 to 46 in the container 80. This is because the image data in FIG. 6 is suitable for the second trained model 132.

[0044] The output results 71, 72, and 73 of these trained models are processed by the score adjustment unit 114 and aggregation unit 115 described above, and the results of this processing are integrated to obtain the ensemble result 74. According to the ensemble result 74, it can be seen that all of the six works 41 to 46 in the container 80 have been detected generally correctly. In other words, in the ensemble result 74, the parts that were correctly detected by the first trained model 131 remain correct without being influenced by results with low detection accuracy such as the output result 73, and the parts that were detected with low accuracy by the first trained model 131 are effectively combined with the correct detection results of the second trained model 132, resulting in a detection result with higher accuracy overall than that of the first trained model 131.

[0045] Unlike FIG. 6, FIG. 7 illustrates a case in which a workpiece 47 exists beneath a workpiece 48. In FIG. 7, if the score is equal to or greater than the threshold, the workpiece is represented by line hatching, while hatching is omitted for other works. The bounding box of the underlying surface is likely to overlap with surrounding bounding boxes, resulting in a low score. The phrase "likely to have a low score" here means that although the score of the bounding box of the underlying surface is equal to or greater than the threshold, it is likely to have a lower score than the surrounding bounding boxes. On the left side of FIG. 7, a workpiece 47 underlying the workpiece 48 is detected. However, by performing processing by the score adjustment unit 114 and the aggregation unit 115, the bounding boxes with low scores are removed, leaving the bounding boxes with high scores. This makes it possible to remove the workpiece 47 so that it is not detected, as shown on the right side of FIG. 7.

[0046] (Processing flow) Next, the flow of processing executed by the information processing device 10 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of processing executed by the information processing device 10.

[0047] In step S101, the data acquisition unit 111 of the information processing device 10 acquires image data obtained by photographing, from above, the container 80 in which the workpieces 41 to 46 to be picked are stored, using a camera (see FIG. 6).

[0048] The processing proceeds to step S102, where the data input unit 112 of the information processing device 10 inputs the image data acquired in step S101 to each of the first trained model 131, the second trained model 132, and the third trained model 133. The first trained model 131, the second trained model 132, and the third trained model 133 are models prepared in advance and stored in the storage device 13. The first trained model 131 is a model trained using large-scale training data 50. The second trained model 132 is a model fine-tuned using small-scale training data 53, which is data selected from the large-scale training data 50 that has a low accuracy rate. The third trained model 133 is a model fine-tuned using small-scale training data 56, which is data selected from the small-scale training data 53 that has a low accuracy rate.

[0049] The process proceeds to step S103, where the weighting unit 113 of the information processing device 10 weights the outputs of the first trained model 131, the second trained model 132, and the third trained model 133 based on the mAP.

[0050] The process proceeds to step S104, where the score adjustment unit 114 of the information processing device 10 calculates overlapping bounding boxes using Soft-NMS and adjusts the scores. Specifically, the score adjustment unit 114 calculates the IoU for the bounding box with the highest score and adjusts the score so that overlaps are eliminated.

[0051] The processing proceeds to step S105, where the aggregation unit 115 of the information processing device 10 calculates the likelihood of the bounding box output by each trained model based on the IoU, score, coordinates, number of trained models, and reliability of the trained models.

[0052] The process proceeds to step S106, where the object detection unit 116 of the information processing device 10 integrates the results of the processes performed in steps S104 and S105 on the output results of each model, and detects the workpieces 41 to 46 to be picked.

[0053] The process proceeds to step S107, where the output unit 117 of the information processing device 10 outputs the detection result of step S106 to the picking control device 20. This allows the desired workpieces 41 to 46 to be picked.

[0054] As described above, the information processing method according to this embodiment includes an input step (S102) of inputting data into each of a plurality of trained models, a calculation step (S103 to S105) of calculating the likelihood of a predetermined event based on the output results of each of the plurality of trained models for the input data, and a recognition step (step S106) of recognizing the event based on the calculation results.

[0055] The processing flow in the flowchart shown in FIG. 8 is an example, and steps may be deleted, new steps may be added, or the processing order may be changed within the scope of the invention.

[0056] (Action and effect) As described above, according to this embodiment, the following advantageous effects can be obtained.

[0057] The information processing device 10 is a device that recognizes a predetermined event by inputting data into a plurality of trained models, and includes at least one processor 11. The processor 11 inputs data into each of the plurality of trained models, calculates the likelihood of the event based on the output results of each of the plurality of trained models for the input data, and recognizes the event based on the calculation results.

[0058] Combining the output results of multiple trained models to calculate the likelihood of a certain event and recognize the event based on the calculation results can improve the reliability of the recognition results. Furthermore, combining multiple trained models allows for combined operation based on the strengths and weaknesses of each model and / or on-site requirements, without relying on the accuracy of any one model. This reduces the risk of the robot arm abnormally stopping due to an error in a picking operation using a robot arm, such as when a product is actually present in a container and the robot arm recognizes that the product is not present. Furthermore, combining multiple trained models eliminates the need to solve all problems with a single model, allowing for problem division and model operation tailored to the problem. This allows for rapid response when a problem occurs by creating an improvised model specialized for the problem and combining it with an existing model, enabling individual responses to each problem as they arise.

[0059] Events detected and recognized by the information processing device 10 include "objects," "animal cries," "fault sounds," "taste," "temperature," "humidity," and "seismic intensity." In the above-described embodiment, an "object" has been described. As an example of a method in which the information processing device 10 detects an object from an image, the information processing device 10 inputs image data to each of a plurality of trained models (e.g., the first trained model 131, the second trained model 132, and the third trained model 133 described above) and weights the scores, which are the outputs of the plurality of trained models for the input image data. Next, the information processing device 10 calculates bounding boxes where the output results of the plurality of trained models overlap, and adjusts the weighted scores. Then, the information processing device 10 retains bounding boxes with high scores and removes bounding boxes with low scores from the overlapping bounding boxes, thereby detecting an object from the image.

[0060] According to the above configuration, by retaining only outputs with high scores (likelihood) from the outputs of multiple trained models, noise can be effectively removed while also handling cases where only one trained model outputs, such as when an item is not detected by the first trained model 131 and the second trained model 132 but is detected by the third trained model 133. This makes it possible to minimize undetected items and retain only plausible outputs. In operations that rely on a single model, such as the conventional technology described above, no matter how much detection accuracy is improved, there is no model that guarantees 100% accuracy, so undetected items are unavoidable. In contrast, in operations that combine multiple trained models, as in the present embodiment, it is possible to retain only plausible outputs, thereby reducing the risk of a product being recognized as not being present in a container when it is actually present.

[0061] Furthermore, in the above description, the first trained model 131, the second trained model 132, and the third trained model 133 are described as models trained to be able to detect identical products, but this is not limited thereto. Each trained model may also be a model trained to be able to detect products of different sizes, shapes, etc. Generally, in "automated picking using image recognition," a master image of the product must be registered in advance. However, by preparing multiple models capable of detecting different products and customizing the combination of these models, it becomes possible to recognize products and pickable locations without a master image. Furthermore, since the products to be recognized typically differ from customer to customer, a model must be tailored for each customer. Therefore, operations that rely on a single model, such as the above-mentioned conventional technology, require a long time to generate a new model. In this regard, a configuration that customizes the combination of multiple pre-prepared models, as in the present embodiment, can accommodate changes in the products to be recognized, thereby meeting customer requirements without the need for the long time required for model generation as in the conventional technology.

[0062] The methods for generating the first trained model 131, the second trained model 132, and the third trained model 133 are not particularly limited, and may be, for example, as follows: The first trained model 131 may be generated by training the large-scale training data 50 for approximately three weeks (see FIG. 5 ). The second trained model 132 may be generated by training, for approximately six hours, small-scale training data 53, which is data selected from the large-scale training data 50 that has a low accuracy rate for the first trained model 131. The third trained model 133 may be generated by training, for approximately two hours, small-scale training data 56, which is data selected from the small-scale training data 53 that has a low accuracy rate for the second trained model 132. The data amounts decrease in the order of the large-scale training data 50, the small-scale training data 53, and the small-scale training data 56. Note that the large-scale training data 50 corresponds to the "first training data," the small-scale training data 53 corresponds to the "second training data," and the small-scale training data 56 corresponds to the "third training data."

[0063] According to the above configuration, the learning time can be significantly reduced by improving the learning method, such as by selecting only a small amount of data with a low accuracy rate and re-learning it.

[0064] Other Embodiments In the above-described embodiment, an example in which image data is input to each trained model has been described. However, the data input to each trained model is not limited to image data. For example, point cloud data may be input to each trained model, or both image data and point cloud data may be input. Here, the terms "image data" and "point cloud data" in the present disclosure will be described. "Image data" is a collection of RGB data in which each color (red, green, and blue) of each pixel is represented by 8 bits, i.e., a value from 0 to 255. "Point cloud data" is a collection of depth data that represents the position from a reference position (e.g., the camera position) to an object using relative three-dimensional coordinates. The information processing device 10 may input both image data and point cloud data to each trained model, calculate the likelihood, and integrate the calculation results to detect a workpiece. A specific example will be described. Here, two trained models are used. For convenience of explanation, one trained model will be referred to as "trained model A" and the other trained model will be referred to as "trained model B." Image data and point cloud data are input into trained model A and trained model B, respectively, and two probabilities are calculated from each trained model: the probabilities of the image data and the probabilities of the point cloud data. These are then integrated for the image data and the point cloud data respectively to produce a single output.

[0065] In this integration method, the scores of the point cloud data are weighted in an increasing direction, and scores other than the scores of the point cloud data (here, the scores of image data as an example) are weighted in a decreasing direction. Point cloud data is more reliable in detecting the presence or absence of an object than image data, etc. Therefore, by weighting the scores of the point cloud data before integration, if the score of the point cloud data is high, the object will be detected or will be easily detected even if the scores of the other data are low. On the other hand, if the score of the point cloud data is low, the object will not be detected or will be difficult to detect even if the scores of the other data are high. This integration method is effective when you want to prevent the object from being detected when the reliability of the score may be questionable (for example, when the score is near a threshold).

[0066] An outline of the integration method in this embodiment will be described with reference to Tables 1 to 3. Table 1 shows the scores when image data and point cloud data are input to trained model A, the scores when image data and point cloud data are input to trained model B, the average values ​​of the respective scores, and the weighting coefficients used in the weighting calculation. The same applies to Tables 2 to 3 and Tables 4 to 9 described below.

[0067] [Table 1]

[0068] In the example shown in Table 1, the output, which is the combined likelihood, is calculated by multiplying the average of each score by a weighting factor, i.e., the output is calculated as {(0.5 + 0.5) / 2 × 0.5 + (0.5 + 0.5) / 2 × 1.5} / 2 = 0.5.

[0069] Compared with Table 1, Table 2 shows cases where the score of point cloud data is higher than the score of image data.

[0070] [Table 2]

[0071] In the example shown in Table 2, the output, which is the combined likelihood, is calculated as {(0.1 + 0.2) / 2 x 0.5 + (0.9 + 0.6) / 2 x 1.5} / 2 = 0.6.

[0072] Compared with Table 1, Table 3 shows cases where the score of the point cloud data is lower than the score of the image data.

[0073] [Table 3]

[0074] In the example shown in Table 3, the output, which is the combined likelihood, is calculated as {(0.9 + 0.6) / 2 x 0.5 + (0.1 + 0.2) / 2 x 1.5} / 2 = 0.3.

[0075] For the calculation results of Tables 2 and 3 above, if the threshold is set to 0.5, the object will be detected according to calculations using the data shown in Table 2, but will not be detected according to calculations using the data shown in Table 3. That is, according to the integration method of this embodiment, as shown in Table 2, if the score of the point cloud data is high, the object will be detected even if the score of the image data is low, whereas as shown in Table 3, if the score of the point cloud data is low, the object will not be detected even if the score of the image data is high.

[0076] Next, a specific example of the integration method in this embodiment will be described with reference to Fig. 9. As shown in Fig. 9, three works, namely works 90 to 92, are placed in a container 80. Two cases will be described for these works 90 to 92: one in which the point cloud data scores are not weighted in an increasing direction and the image data scores are not weighted in a decreasing direction, and one in which these weightings are applied. First, the case in which no weighting is applied will be described with reference to Tables 4 to 6. Table 4 shows the data for work 90, Table 5 shows the data for work 91, and Table 6 shows the data for work 92.

[0077] [Table 4]

[0078] [Table 5]

[0079] [Table 6]

[0080] As shown in Tables 4 to 6, for each of works 90 to 92, the weighting coefficient is 1 because no weighting is applied to the point cloud data score. Also, the weighting coefficient is 1 because no weighting is applied to the image data score. As shown in Table 4, the likelihood of work 90 is calculated as {(0.9 + 0.9) / 2 × 1 + (0.7 + 0.8) / 2 × 1} / 2 = 0.825. Similarly, as shown in Table 5, the likelihood of work 91 is calculated as {(0.7 + 0.8) / 2 × 1 + (0.2 + 0.1) / 2 × 1} / 2 = 0.45. Similarly, as shown in Table 6, the likelihood of work 92 is calculated as {(0.7 + 0.7) / 2 × 1 + (0.4 + 0.2) / 2 × 1} / 2 = 0.5.

[0081] In the examples shown in Tables 4 to 6, when the threshold value is set to 0.5, the information processing device 10 detects the workpiece 90 and the workpiece 92.

[0082] Next, a case where an increasing weight is assigned to the score of point cloud data and a decreasing weight is assigned to the score of image data will be described with reference to Tables 7 to 9. Table 7 shows data for work 90, Table 8 shows data for work 91, and Table 9 shows data for work 92. The only difference between Tables 4 to 6 and Tables 7 to 9 is the value of the weighting coefficient.

[0083] [Table 7]

[0084] [Table 8]

[0085] [Table 9]

[0086] As shown in Tables 7 to 9, for each of works 90 to 92, the weighting coefficient is 1.5 because the point cloud data score is weighted in an increasing direction. Furthermore, the weighting coefficient is 0.5 because the image data score is weighted in a decreasing direction. As shown in Table 7, the likelihood of work 90 is calculated as {(0.9 + 0.9) / 2 × 0.5 + (0.7 + 0.8) / 2 × 1.5} / 2 = 0.7875. Similarly, as shown in Table 8, the likelihood of work 91 is calculated as {(0.7 + 0.8) / 2 × 0.5 + (0.2 + 0.1) / 2 × 1.5} / 2 = 0.3. Similarly, as shown in Table 9, the likelihood of work 92 is calculated as {(0.7 + 0.7) / 2 × 0.5 + (0.4 + 0.2) / 2 × 1.5} / 2 = 0.4.

[0087] In the examples shown in Tables 7 to 9, when the threshold value is set to 0.5, the information processing device 10 detects only the workpiece 90.

[0088] Comparing the detection results in Tables 4 to 6 with the detection results in Tables 7 to 9, it can be seen that the workpiece 92 is detected when no weighting is performed, but is not detected when weighting is performed. In this way, by performing an increasing weighting on the point cloud data score and a decreasing weighting on the image data score, it is possible to achieve a result in which the workpiece 92 is not detected when the point cloud data score is low, even if the image data score is high. The likelihood of the workpiece 92 when no weighting is performed is "0.5," which exactly overlaps with the threshold value. In this way, the integration method of this embodiment makes it possible to prevent the object from being detected when there is a possibility that the reliability of the score may be in doubt.

[0089] In the above description, an example of an integration method is described in which image data and point cloud data are input to trained model A and trained model B, respectively, and two probabilities, the probabilities of the image data and the probabilities of the point cloud data, are calculated (averaged) from each trained model, and then integrated for each of the image data and the point cloud data to produce a single output. However, this is not limiting. The integration method may also be to calculate a single probabilities by combining the image data and the point cloud data from each trained model and integrate the calculation results. In this case, the above-described weighting may be applied to each score before combining the image data and the point cloud data, i.e., before averaging.

[0090] In the above-mentioned integration method, a method of multiplying a weighting coefficient greater than 1 is described as a way to relatively increase the score of the point cloud data, but this is not limited to this, and any weighting method can be used as long as the score of the point cloud data is relatively high.

[0091] The integration method in this embodiment may be expressed as follows. That is, the data input to the multiple trained models may include at least point cloud data. The processor 11 may input the point cloud data to each of the multiple trained models, and may assign an increasing weight to the scores of the point cloud data, which are the outputs of each of the multiple trained models, and may assign a decreasing weight to the scores of data other than the point cloud data.

[0092] Furthermore, although the above-described first trained model 131, second trained model 132, and third trained model 133 are each described as models that perform an object detection task, the models to be combined are not limited to models that perform the same task. Models that perform different tasks may be combined. For example, as an example of a combination, a model that performs an object detection task may be combined with a model that performs an appearance detection task. The model that performs an appearance detection task is a model used for visual inspection of the recognition surface, such as dashed lines and bottoms. By combining models that perform different tasks in this way, operation that addresses individual issues becomes possible. As another example of a combination, in addition to the combination of a model that performs an object detection task with a model that performs an appearance detection task, a model that performs a motion anomaly detection task may also be combined. The model that performs a motion anomaly detection task is a model that detects motion anomalies, such as the success or failure of adsorption, during adsorption operations of a robot arm.

[0093] [Software implementation example] The functions of the information processing device 10 can be realized by an information processing program for causing a computer to function as the information processing device 10, and a program for causing a computer to function as each control block of the information processing device 10.

[0094] In this case, the information processing device 10 includes a computer having at least one device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The computer executes the program to realize each function described in each embodiment.

[0095] The program may be stored non-transitory on one or more computer-readable recording media. The recording media may or may not be included in the information processing device 10. In the latter case, the program may be supplied to the information processing device 10 via any wired or wireless transmission medium.

[0096] In addition, some or all of the functions of each control block can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each control block is formed is also included in the scope of the present disclosure. In addition, the functions of each control block can also be realized by, for example, a quantum computer.

[0097] Furthermore, each process described in each embodiment may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the information processing device 10 or on another device (for example, an edge computer or a cloud server).

[0098] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0099] 100 Information Processing Systems 10. Information processing equipment 11 processors 131 First trained model 132 Second trained model 133 Third trained model

Claims

1. An information processing device that recognizes a predetermined event by inputting data into a plurality of trained models, at least one processor; The processor: inputting the data into each of the plurality of trained models; Calculating the likelihood of the event based on the output results of each of the plurality of trained models for the input data; Recognizing the event based on the calculation result. Information processing device.

2. The data includes at least one of image data and point cloud data, which is a collection of depth data that represents the position from a reference position to an object in three-dimensional coordinates; The processor: inputting the data into each of the plurality of trained models; Weighting the scores that are outputs of the plurality of trained models for the input data; The information processing device according to claim 1 .

3. The data includes point cloud data, which is a collection of depth data that represents positions from a reference position to an object in three-dimensional coordinates; The processor: inputting the point cloud data into each of the plurality of trained models; weighting the scores of the point cloud data, which are outputs of each of the plurality of trained models, in an increasing direction, and weighting the scores of data other than the point cloud data in a decreasing direction; The information processing device according to claim 1 .

4. The processor calculates a bounding box where the output results of the plurality of trained models overlap, and adjusts the weighted score. The information processing device according to claim 2 .

5. the data includes at least image data; The processor: Among the overlapping bounding boxes, the bounding box with the higher score is kept and the bounding box with the lower score is removed; Detecting objects in an image The information processing device according to claim 4 .

6. the plurality of trained models include a first trained model, a second trained model, and a third trained model; the first trained model is a model trained using first training data, the second trained model is a model trained using second training data selected from the first training data for which the accuracy rate of the first trained model is low, the third trained model is a model trained using third training data selected from the second training data, data for which the second trained model has a low accuracy rate; The data amount decreases in the order of the first learning data, the second learning data, and the third learning data. The information processing device according to claim 1 .

7. The plurality of trained models includes at least two models for different tasks. The information processing device according to any one of claims 1 to 6.

8. An information processing method executed by one or more information processing devices, An input step for inputting data into each of the multiple trained models; a calculation step of calculating a likelihood of a predetermined event based on the output results of each of the plurality of trained models for the input data; and recognizing the event based on the calculation result. Information processing methods.

9. An information processing program that causes a computer to execute an input process for inputting data into each of a plurality of trained models, a calculation process for calculating the likelihood of a predetermined event based on the output results of each of the plurality of trained models for the data, and a recognition process for recognizing the event based on the calculation results.

Images