Processing device, processing system, processing method, program, and storage medium

Abstract

The present invention provides a processing device, a processing system, a processing method, a program, and a storage medium that can select untrained data suitable for updating a model. [Solution] The processing device according to the embodiment inputs a plurality of untrained data that have not been used for training into a trained first model, and obtains a plurality of output results output from the first model. The processing device calculates a plurality of scores for each of the plurality of untrained data using the plurality of output results. The processing device divides the plurality of untrained data into a plurality of groups by clustering the feature data extracted from each of the plurality of untrained data. The processing device selects one or more of the untrained data from each of the plurality of groups. In the selection of the untrained data, the number of untrained data selected from each of the plurality of groups is determined according to the score of the untrained data included in each of the plurality of groups.

Description

Technical Field

[0001] Embodiments of the present invention relate to a processing device, a processing system, a processing method, a program, and a storage medium.

Background Art

[0002] Machine-learned models are used in various fields. In machine learning, a model is learned using a large number of learning data. When data similar to the learning data is input to the model, an output result with good accuracy can be obtained. On the other hand, when unlearned data that is not similar to the learning data is input to the model, the accuracy of the output result may decrease. In order to improve the accuracy of the output result of the model, it is desirable to update the model using unlearned data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the embodiments of the present invention is to provide a processing device, a processing system, a processing method, a program, and a storage medium that can select unlearned data suitable for updating a model.

Means for Solving the Problems

[0005] The processing device according to the embodiment inputs a plurality of untrained data that have not been used for training into a trained first model, and obtains a plurality of output results output from the first model. The processing device calculates a plurality of scores for each of the plurality of untrained data using the plurality of output results. The processing device divides the plurality of untrained data into a plurality of groups by clustering the feature data extracted from each of the plurality of untrained data. The processing device selects one or more of the untrained data from each of the plurality of groups. In the selection of the untrained data, the number of untrained data selected from each of the plurality of groups is determined according to the score of the untrained data included in each of the plurality of groups. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a schematic diagram showing a processing system according to an embodiment. [Figure 2] Figure 2 is a flowchart showing the processing method according to the embodiment. [Figure 3] Figures 3(a) and 3(b) are schematic diagrams showing the inputs and outputs of the first model in the first embodiment. [Figure 4] Figure 4 is a schematic diagram showing the result of feature data extraction in the first embodiment. [Figure 5] Figure 5 is a schematic diagram showing the clustering results in the first embodiment. [Figure 6] Figure 6 is a schematic diagram showing the method for selecting untrained data in the first embodiment. [Figure 7] Figure 7 is a schematic diagram showing the method for selecting untrained data in the first embodiment. [Figure 8] Figure 8 is a schematic diagram showing the method for selecting untrained data in the first embodiment. [Figure 9] Figure 9 is a schematic diagram illustrating a graphical user interface. [Figure 10] Figure 10 is a schematic diagram illustrating a graphical user interface. [Figure 11] Figure 11 is a perspective view showing an example of a handling system according to the first embodiment. [Figure 12] Figures 12(a) and 12(b) are schematic diagrams showing the images input to the second embodiment. [Figure 13] Figure 13 is a schematic diagram representing the hardware configuration. [Modes for carrying out the invention]

[0007] The embodiments of the present invention will be described below with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes of the parts, etc., are not necessarily the same as those of reality. Furthermore, even when representing the same part, the dimensions and ratios may be represented differently in the drawings. In this specification and each drawing, elements similar to those already described are denoted by the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] Figure 1 is a schematic diagram showing a processing system according to an embodiment. The processing system 1 according to this embodiment includes a processing unit 10, an input device 20, a display device 30, and a storage device 40.

[0009] The storage device 40 stores the untrained data 41 and the first model M1. The first model M1 has been trained using a large amount of training data 42. Here, "untrained data" refers to data that was not used during the training of the first model M1. For example, untrained data is data that has different properties or different information from the training data. Examples of "untrained data" include data that falls outside the distribution of the training data, data generated under different circumstances or conditions than the training data, and data that can provide new information to the first model M1.

[0010] The processing unit 10 selects some unlearned data 41 from a plurality of unlearned data 41 stored in the storage device 40 that are suitable for updating the first model M1. The input device 20 is used by the user to input information to the processing unit 10. The processing unit 10 displays the information on the display device 30 for the user.

[0011] Specifically, the processing unit 10 has the functions of an acquisition unit 11, a calculation unit 12, an extraction unit 13, a splitting unit 14, and a selection unit 15. The acquisition unit 11 acquires multiple unlearned data 41 from the storage device 40. The acquisition unit 11 inputs the unlearned data 41 into the first model M1 and acquires the output results from the first model M1. Multiple unlearned data 41 are sequentially input into the first model M1, and multiple output results are acquired.

[0012] The calculation unit 12 calculates a score for one untrained data from its output result. The score is an indicator of the accuracy of the first model M1 for the untrained data. The calculation unit 12 uses multiple output results to calculate multiple scores for multiple untrained data. For example, a higher score indicates that the first model M1 is producing favorable results for that untrained data. If the first model M1 outputs a value that corresponds to the score, the calculation unit 12 may use the value output from the first model M1 to calculate the score. Alternatively, the calculation unit 12 may use the value output from the first model M1 as the score. Here, an embodiment will be described in which a higher score indicates higher output accuracy of the model for that untrained data.

[0013] The extraction unit 13 extracts feature data from each of the plurality of unlearned data 41. For example, the extraction unit 13 inputs the unlearned data 41 to the feature extractor 43 stored in the storage device 40. When the unlearned data 41 is input, the feature extractor 43 outputs feature data corresponding to the unlearned data 41. The extraction unit 13 sequentially inputs the plurality of unlearned data 41 to the feature extractor 43 to extract a plurality of feature data from the plurality of unlearned data 41 respectively. The feature extractor 43 includes a learned neural network. Preferably, the feature extractor 43 includes a convolutional neural network (CNN). The specific configuration of the feature extractor 43 can be arbitrarily selected. The feature extractor 43 is used to extract a plurality of feature data respectively from the plurality of unlearned data 41 by a common means.

[0014] The partitioning unit 14 clusters the plurality of feature data. As a result, the plurality of unlearned data 41 are partitioned into a plurality of groups. For clustering, hierarchical clustering (such as the ward method), non-hierarchical clustering (such as the K-means method), density-based clustering (DBSCAN, Gaussian Mixture Model, etc.), clustering using deep learning (DNN, SOM, AE or combinations thereof, etc.) can be used.

[0015] The selection unit 15 selects some of the plurality of unlearned data 41 based on the plurality of scores calculated by the calculation unit 12 and the result of clustering by the partitioning unit 14. At this time, the number of unlearned data 41 selected in each group is determined according to the scores of the unlearned data 41 included in each group. For example, relatively few unlearned data are selected from the group with a high representative value of the score, and relatively many unlearned data are selected from the group with a low representative value of the score.

[0016] As a specific example, by clustering, a plurality of unlearned data 41 are divided into a plurality of groups. The plurality of groups include a first group and a second group. A plurality of first unlearned data included in the plurality of unlearned data 41 are divided into the first group. A plurality of second unlearned data included in the plurality of unlearned data 41 are divided into the second group. When the average value of the scores of the plurality of first unlearned data is higher than a predetermined first threshold value and the average value of the scores of the plurality of second unlearned data is lower than the first threshold value, the selection unit 15 selects n unlearned data 41 from the first group and m unlearned data 41 from the second group. "m" is a numerical value larger than "n". For example, "m" may be 1, and "n" may be 2 or more.

[0017] Clustering may be used to determine "n". The division unit 14 clusters the feature data of the plurality of second unlearned data included in the second group. Thereby, the plurality of second unlearned data are divided into a plurality of sub-groups. The selection unit 15 selects one or more (for example, m) second unlearned data from each of the plurality of sub-groups.

[0018] The "representative value" is the average value, median value, or most frequent value of the plurality of scores. The first threshold value may be set by the user or may be set by the processing device 10. For example, the processing device 10 may set the average value, median value, or most frequent value of the scores of all the unlearned data 41 as the first threshold value.

[0019] The selection unit 15 stores the selected unlearned data 41 in the storage device 40. Alternatively, as shown in FIG. 1, the processing device 10 may have functions as the expansion unit 16 or the learning unit 17. In that case, further processing is executed using the selected unlearned data 41.

[0020] The selection unit 15 displays the selected unlearned data on the display device 30. The user provides instruction to the unlearned data displayed on the display device 30 using the input device 20. The expansion unit 16 receives the instruction data input by the user. The expansion unit 16 expands the selected unlearned data using the instruction data. The number of unlearned data increases through this expansion.

[0021] The learning unit 17 updates the first model M1 using the selected unlearned data 41. During the update, the first model M1 is further trained using the selected unlearned data 41. Alternatively, the first model M1 may be retrained using the selected unlearned data 41 and the trained data 42. To shorten the time required to update the first model M1, it is preferable to perform additional training. If the unlearned data is expanded, the first model M1 is updated using the expanded unlearned data as well.

[0022] Figure 2 is a flowchart showing the processing method according to the embodiment. First, the acquisition unit 11 inputs multiple untrained data 41 to the first model M1 and obtains multiple output results from the first model M1 (step St1). The calculation unit 12 uses these multiple output results to calculate multiple scores for each of the multiple untrained data 41 (step St2). The extraction unit 13 extracts feature data from each of the multiple untrained data (step St3). The splitting unit 14 divides the multiple untrained data 41 into multiple groups by clustering the multiple feature data (step St4). The selection unit 15 selects one or more untrained data from each of the multiple groups (step St5). Subsequently, the selected untrained data may be expanded (step St6). The first model M1 may be updated using the selected untrained data (step St7).

[0023] The advantages of the embodiment will be explained. When updating a trained model, it is desirable that the update time be short. Updating a model requires human intervention with untrained data. The more data that needs to be trained, the greater the time required for the update and the greater the burden on the human involved. To shorten the update time, it is preferable to select data suitable for the update from the data that has not yet been trained. However, it is not easy for a human to appropriately select data suitable for the update.

[0024] According to this embodiment, the acquisition unit 11 inputs multiple unlearned data 41 to the trained first model M1 and acquires multiple output results output from the first model M1. The calculation unit 12 then uses the multiple output results to calculate multiple scores for each of the multiple unlearned data 41. As described above, the score is an indicator of the accuracy of the first model M1 for the unlearned data. For example, a low score means that the unlearned data contains information unknown to the first model M1. The score allows us to examine the necessity of training the first model M1 for each unlearned data.

[0025] Furthermore, the extraction unit 13 extracts multiple feature data by inputting each of the multiple untrained data 41 into the feature extractor 43. By inputting multiple untrained data 41 into a common feature extractor 43 and obtaining their feature data, the characteristics of each untrained data from the computer's perspective can be examined. The splitting unit 14 divides the multiple untrained data 41 into multiple groups by clustering these feature data. As a result, the multiple untrained data 41 are divided into multiple groups according to their characteristics from the computer's perspective.

[0026] The selection unit 15 then selects some of the untrained data 41 from each group. One group is a collection of data with high similarity. For example, even if a large number of untrained data 41 are selected from one group, the accuracy of the first model M1 with respect to the untrained data 41 included in another group is unlikely to improve. By selecting some of the untrained data 41 from each group, a wider variety of untrained data 41 can be selected.

[0027] In this process, the number of unlearned data points 41 selected from each group is determined according to the scores of the unlearned data points 41 included in each group. For example, more unlearned data points 41 are selected from groups of unlearned data points 41 with low scores, while fewer unlearned data points 41 are selected from groups of unlearned data points 41 with high scores. In other words, a low score means that there is a high need to learn about that unlearned data point 41. Therefore, unlearned data points 41 with a high need for learning can be selected efficiently.

[0028] According to this embodiment, diverse and highly necessary untrained data can be selected. For example, by using the selected untrained data for updating, a first model that can handle a wider variety of data and has high accuracy can be obtained.

[0029] It is preferable that the untrained data 41 be expanded. Expansion increases the number of untrained data 41 available for training. As a result, the accuracy of the updated first model M1 can be improved.

[0030] The method for selecting untrained data 41 from each group can be arbitrarily set. For example, the selection unit 15 may select untrained data 41 in each group based on a score, or it may select untrained data 41 based on the features of each untrained data. Preferably, the selection unit 15 selects the untrained data 41 in each group that is closest to the representative value of the features of the multiple untrained data 41. The representative value may be the mean or the centroid. The untrained data 41 closest to the representative value is the data that best represents the features of the multiple untrained data 41 belonging to that group. By training the first model M1 using the untrained data 41 closest to the representative value, the accuracy of the first model M1 can be further improved.

[0031] Examples of the embodiments will be described below.

[0032] (First embodiment) In the first embodiment, image data is input to the first model M1. The first model M1 recognizes the region in the input image data in which a predetermined object is depicted and outputs the recognition result. The first model M1 may further output a score related to the recognition result. Preferably, the first model M1 includes a CNN.

[0033] Figures 3(a) and 3(b) are schematic diagrams showing the inputs and outputs of the first model in the first embodiment. For example, the image IMG shown in Figure 3(a) is input to the first model M1. As shown in Figure 3(b), the first model M1 recognizes the region in the image IMG that contains one object O1 and outputs the recognition result and score.

[0034] Figure 4 is a schematic diagram showing the result of feature data extraction in the first embodiment. The extraction unit 13 sequentially inputs multiple untrained data 41 into the feature extractor 43 and extracts multiple feature data. For example, the feature extractor 43 includes a CNN. As a result, as shown in Figure 4, feature data FD is extracted from each untrained data 41.

[0035] Figure 5 is a schematic diagram showing the clustering results in the first embodiment. The splitting unit 14 clusters the multiple feature data FDs. As a result, as shown in Figure 5, for example, the multiple feature data FDs are divided into six groups Gr1 to Gr6.

[0036] Figures 6 to 8 are schematic diagrams illustrating the method for selecting untrained data in the first embodiment. The selection unit 15 calculates representative scores for each group, as shown in Figure 6. In Figure 6, the numbers listed next to each group are examples of scores. Furthermore, the selection unit 15 refers to a first threshold for the representative scores. For example, the first threshold is 0.5. In this case, the scores of groups Gr1 to Gr3 are greater than the first threshold. The scores of groups Gr4 to Gr6 are less than the first threshold.

[0037] The selection unit 15 further performs clustering on the untrained data included in groups Gr4 to Gr6. As a result, as shown in Figure 7, the feature data FD included in group Gr4 is divided into two subgroups, Gr4a and Gr4b. The feature data FD included in group Gr5 is divided into three subgroups, Gr5a to Gr5c. The feature data FD included in group Gr6 is divided into three subgroups, Gr6a to Gr6c.

[0038] As shown in Figure 8, the selection unit 15 selects one feature data FD from each of the groups Gr1 to Gr3. The selection unit 15 also selects one feature data FD from each of the subgroups Gr4a, Gr4b, Gr5a to Gr5c, and Gr6a to Gr6c. As a result, one feature data FD is selected from each of the groups Gr1 to Gr3. Two or more feature data FDs are selected from each of the groups Gr4 to Gr6. The selection unit 15 selects the untrained data 41 from which the selected feature data FDs were extracted as the untrained data 41 to be used for updating the first model M1.

[0039] Figures 9 and 10 are schematic diagrams illustrating a graphical user interface. The processing unit 10 displays, for example, the graphical user interface (GUI) 100 shown in Figure 9 on the display device 30. The GUI 100 includes selection areas 110, 120, and 130.

[0040] In the selection area 110, candidate untrained data to be used for updating are selected. For example, the user clicks the pull-down icon 111 using the input device 20. This displays a list of training data sets available for updating. One training data set contains multiple untrained data. The user selects one of the displayed training data sets. After selecting a training data set, the user clicks icon 112. In response to the click of icon 112, the processing unit 10 executes the process shown in Figure 2. This selects some of the untrained data 41 from the multiple untrained data 41 included in the selected training data set.

[0041] In the selection area 120, the image to be annotated is selected. The user clicks either icon 121 or 122. In the display area 123, one untrained data point 41 is displayed from the training data set selected in the selection area 110. Depending on the click of icon 121 or 122, the untrained data point 41 displayed in the display area 123 changes. Once the untrained data point 41 to be annotated is displayed in the display area 123, the user clicks icon 124. When icon 124 is clicked, an annotation window is displayed. In that window, the user masks a portion of the image. When masking, the user can mask areas according to the features of the object. For example, as shown in Figure 9, the entire outline of one object in the image may be masked, or as shown in Figure 10, only a portion of the object may be masked. The mask applied by the user is used as annotation for the image. The masked untrained data point is displayed in the display area 125. The user confirms the untrained data point 41 displayed in the display area 125. If the displayed unlearned data 41 is properly annotated, the user clicks icon 126. This links and saves the user's annotations to the unlearned data 41.

[0042] In the selection area 130, the unlearned data 41 to be augmented is selected. When the user clicks the pull-down icon 131, a list of annotated unlearned data 41 is displayed. The user selects one of the unlearned data 41 from the displayed list. After selecting the unlearned data 41, the user clicks icon 132. In response to the click of icon 132, the processing unit 10 performs data augmentation on the selected unlearned data 41.

[0043] For example, the extension unit 16 saves the annotated area as the object area containing the object. The extension unit 16 saves the area other than the annotated area as the background area containing the background. The extension unit 16 performs image processing on one or both of the object area and the background area. The image processing is one or more selected from the group consisting of, for example, contrast change, brightness change, resizing, position change, noise addition, rotation, inversion, channel shift, and pixel value rescaling. After image processing, the object area and the background area are combined to obtain the extended untrained data.

[0044] The learning unit 17 updates the first model M1 using the unlearned data selected by the selection unit 15 and the unlearned data expanded by the expansion unit 16.

[0045] Figure 11 is a perspective view showing an example of a handling system according to the first embodiment. The first embodiment is applicable to a robot-based handling system. The handling system 200 shown in Figure 11 performs object handling using a pre-trained first model M1.

[0046] The handling system 200 includes a handling robot 210, a camera 220, and a processing unit 230. The handling robot 210 includes a robot arm 211 and a base 212. The robot arm 211 includes a plurality of links 211a and a plurality of rotation axes 211b. The links 211a are connected to each other by the rotation axes 211b. In the illustrated example, the robot arm 211 is a vertical articulated type. The robot arm 211 may also be a horizontal articulated type.

[0047] The movement of each rotation axis 211b changes the position and orientation (angle) of the tip of the robot arm 211. Preferably, the tip of the robot arm 211 has six degrees of freedom. The gripping section 215 is attached to the tip of the robot arm 211. In the illustrated example, the gripping section 215 includes a suction mechanism 216 and a clamping mechanism 217.

[0048] The suction mechanism 216 grasps an object by suction. The suction mechanism 216 includes one or more suction pads 216a. With the suction pads 216a in contact with the object, the inside of the suction pads 216a is depressurized by a depressurization device (not shown). As a result, the object is attracted to the suction pads 216a. The number of suction pads 216a may be less or more than the example shown.

[0049] The clamping mechanism 217 grips an object by clamping. The clamping mechanism 217 includes a plurality of rod-shaped support parts 217a. The object is clamped and gripped by the plurality of support parts 217a. The clamping mechanism 217 may include more support parts 217a than shown in the illustrated example. The support parts 217a may be configured to be finger-like, including one or more joints.

[0050] The gripping section 215 further includes a switching mechanism 218. The suction mechanism 216 and the clamping mechanism 217 are connected to the switching mechanism 218. The switching mechanism 218 rotates the suction mechanism 216 and the clamping mechanism 217. By rotating the suction mechanism 216 and the clamping mechanism 217, the mechanism used to grip an object can be switched.

[0051] The gripping section 215 may include only one of the suction mechanism 216 or the clamping mechanism 217, not limited to the illustrated example. In that case, the switching mechanism 218 is unnecessary.

[0052] The robot arm 211 is further equipped with a sensor 213. The sensor 213 is capable of detecting one or more selected from the group consisting of the load applied to the gripping part 215, the torque applied to the gripping part 215, the acceleration of the gripping part 215, and the angular velocity of the gripping part 215. For example, the sensor 213 includes one or more selected from the group consisting of a force sensor, an acceleration sensor, and an angular velocity sensor.

[0053] Two containers, C1 and C2, are placed near the handling robot 210. The handling robot 210 grasps object O contained in container C1 and transports it to container C2.

[0054] Camera 220 is provided for taking pictures inside container C1. For example, camera 220 includes one or more selected from an image sensor and a depth sensor. Camera 220 may be fixed above container C1 or attached to a handling robot 210.

[0055] The processing unit 230 acquires image data obtained by the camera 220. The processing unit 230 refers to the trained first model M1. When the first model M1 receives image data as input, it recognizes the objects in the image. The processing unit 230 uses the recognition results output from the first model M1 to generate a plan for transporting object O. The processing unit 230 causes the handling robot 210 to transport the object according to the generated plan.

[0056] The update to the first model M1 will enable more accurate recognition of objects that were not previously trained. As a result, objects can be grasped more reliably and stably, improving handling accuracy.

[0057] (Second example) In the second embodiment, the first model is used as an anomaly detection model. When image data is input to the first model, the first model outputs a result indicating whether the input image is normal or not. "Anomaly" refers to data that deviates significantly from the statistical or characteristic patterns of data considered normal, data with a very low observation frequency in the overall data, data that does not meet specific criteria or conditions, etc. To improve detection accuracy, the first model preferably includes a CNN.

[0058] Figures 12(a) and 12(b) are schematic diagrams showing the images input to the second embodiment. Figures 12(a) and 12(b) show photographs of clothing. Image IMG1 in Figure 12(a) shows normal clothing without defects such as fraying or stains. Image IMG2 in Figure 12(b) shows abnormal clothing with a stain defect DF. When the first model M1 receives image data as input, it outputs a result indicating whether the image is normal or abnormal.

[0059] For example, the first model M1 outputs a score indicating abnormality. The closer the score is to 0, the more normal the image data is. The first model M1 may also output a probability vector for [normal, abnormal], and the classification result of normal or abnormal may be indicated by the probability vector.

[0060] In the second embodiment, the multiple untrained data are all image data. The acquisition unit 11 sequentially inputs the multiple untrained data into the first model M1 and acquires multiple output results. The calculation unit 12 uses these output results to calculate multiple scores for each of the multiple untrained data. As described above, if a score indicating abnormality is output from the first model M1, the output score may be used as is as the score for the untrained data. If a probability vector is output, the value of the component indicating abnormality may be used as the score.

[0061] After the score is calculated, multiple untrained data 41 are sequentially input to the feature extractor 43, similar to the first embodiment, and multiple feature data are extracted. These feature data FDs are clustered, and a representative score for each group is calculated. Feature data belonging to groups with scores greater than the first threshold are further clustered. Then, one feature data FD is selected from each group or subgroup. The untrained data 41 from which the selected feature data was extracted is selected as the untrained data 41 to be used to update the first model M1.

[0062] The selected untrained data 41 may be expanded. For example, the expansion unit 16 expands the selected untrained data 41 by randomly adding defects to it. The defects are one or more selected from, for example, scratches, stains, and holes. A generative model for expanding the data may be prepared in advance and used. The generative model randomly adds points or defects to the input image. The expansion unit 16 acquires the image output from the generative model as expanded untrained data 41.

[0063] As described above, the second embodiment is applicable to the visual inspection of products. Alternatively, the second embodiment may be applied to a robotic handling system, similar to the first embodiment. For example, when an image of an object is acquired by the camera 220, the first model M1 detects whether the object in the image is normal or not. This allows for simultaneous handling of the object and visual inspection of the object.

[0064] (Third embodiment) In the third embodiment, time-series data is input to the first model M1. That is, in the third embodiment, time-series changing data such as sensor values, temperature history, vibration history, heart rate information, and device operation logs are used instead of image data. For example, data collected at regular intervals from sensors attached to the target device, or various log information transmitted periodically from a predetermined terminal, are stored in the storage device 40 as unlearned data 41. The first model M1 may include a network such as a recurrent neural network (RNN), LSTM (Long Short-Term Memory), or GRU (Gated Recurrent Unit).

[0065] The acquisition unit 11 acquires multiple untrained data 41 and sequentially inputs these untrained data 41 into the first model M1. The acquisition unit 11 acquires output results, which are the inference or analysis results of the model. The calculation unit 12 uses the acquired output results to calculate multiple scores for the multiple untrained data 41. The scores may be set to indicate the prediction error or degree of anomaly of the first model M1 for the time series data. A higher score may be defined as indicating time series data that deviates from the normal pattern, while a lower score may be defined as including unknown behavior.

[0066] The extraction unit 13 sequentially inputs the time-series data contained in each untrained data 41 to the feature extractor 43 and extracts feature data. The feature extractor 43 may include, for example, an autoencoder including an RNN or LSTM layer. The splitting unit 14 performs clustering on the multiple feature data obtained by the extraction unit 13. For clustering, methods such as the K-means method described above may be used, or hierarchical clustering using dynamic time warping (DTW) distance may be performed.

[0067] The selection unit 15 combines the scores calculated by the calculation unit 12 with the clustering results from the splitting unit 14 to select unlearned data 41 for each group. For example, if a higher score indicates that the time series data is abnormal, a relatively large number of unlearned data 41 will be selected from the group with a low representative score (high degree of unknown), and a relatively small number of unlearned data 41 will be selected from the group with a high representative score (high degree of known).

[0068] The extension unit 16 extends the selected unlearned data 41. For example, if the selected unlearned data 41 is taught (labeled) by the user, the user performs annotation work such as adding correct information at each point in time of the time series data or tagging a certain range as anomaly. The extension unit 16 extends the unlearned data by, for example, adding noise locally or cutting out a portion of the data and shifting the time. Alternatively, a portion of the time series may be stretched or compressed.

[0069] The third embodiment is applicable to a handling system, similar to the first embodiment. For example, in the handling system 200 shown in Figure 11, the rotation angle of each rotation axis 211b is acquired as time-series data. The first model M1 outputs the posture of the robot arm 211 at a predetermined timing in response to the input of time-series data of rotation angles. For example, the first model M1 outputs the posture of the robot arm 211 at the timing when the gripping part 215 grips an object. The posture is represented by a combination of the rotation angles of each rotation axis 211b.

[0070] By updating the first model M1, it will be able to output a more appropriate orientation even for objects that were not previously trained. As a result, objects can be grasped more reliably and stably, improving handling accuracy.

[0071] Figure 13 is a schematic diagram representing the hardware configuration. As the processing unit 10, for example, the computer 90 shown in Figure 13 is used. The computer 90 includes a processing circuit 91, ROM 92, RAM 93, storage device 94, input interface 95, output interface 96, and communication interface 97.

[0072] ROM92 stores programs that control the operation of computer 90. ROM92 contains the programs necessary for computer 90 to perform each of the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are loaded.

[0073] The processing circuit 91 includes an arithmetic processing unit such as a CPU or GPU. The processing circuit 91 uses RAM 93 as work memory and executes a program stored in at least one of ROM 92 or storage device 94. During program execution, the processing circuit 91 controls each component via the system bus 98 and performs various processes.

[0074] The memory device 94 stores data necessary for program execution and data obtained through program execution.

[0075] The input interface (I / F) 95 allows connection between the computer 90 and the input device 20. The input I / F 95 is, for example, a serial bus interface such as USB. The processing circuit 91 can read various data from the input device 20 via the input I / F 95.

[0076] The output interface (I / F) 96 can connect the computer 90 and the display device 30. The output I / F 96 is a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI®). The processing circuit 91 can send data to the display device 30 via the output I / F 96 and display a GUI on the display device 30.

[0077] The communication interface (I / F) 97 can connect the computer 90 to a server 97a located outside the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The processing circuit 91 can read various data from the server via the communication I / F 97.

[0078] The storage device 94 includes one or more selected from Hard Disk Drives (HDDs) and Solid State Drives (SSDs). The storage device 94 may also be used as the storage device 40.

[0079] The input device 20 includes one or more selected from a mouse, keyboard, microphone (voice input), and touchpad. The display device 30 includes one or more selected from a monitor and projector. A device that combines the functions of both the input device 20 and the display device 30, such as a touch panel, may also be used.

[0080] Each process performed by the processing unit 10 may be implemented by a single computer 90, or by the cooperation of multiple computers 90.

[0081] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.

[0082] For example, data on a recording medium is read by a computer (or embedded system). The recording format (storage format) on the recording medium is arbitrary. For example, a computer reads a program from the recording medium and causes the CPU to execute instructions based on this program. The acquisition (or reading) of the program by the computer may be done via a network.

[0083] Embodiments of the present invention include the following features. (Feature 1) Multiple untrained data points that were not used in training are input to the first trained model, and multiple output results are obtained from the first model. Using the above-mentioned multiple output results, calculate the above-mentioned multiple scores for the above-mentioned untrained data, By clustering the feature data extracted from each of the aforementioned multiple untrained data sets, the aforementioned multiple untrained data sets are divided into multiple groups. Select one or more of the unlearned data from each of the aforementioned groups, A processing device in which, in the selection of unlearned data, the number of unlearned data selected from each of the plurality of groups is determined according to the score of the unlearned data included in each of the plurality of groups. (Feature 2) The aforementioned multiple groups include the first group and the second group, The processing apparatus according to Feature 1, wherein a plurality of first unlearned data included in the plurality of unlearned data are divided into a first group, a plurality of second unlearned data included in the plurality of unlearned data are divided into a second group, and when the representative value of the score of the plurality of first unlearned data is higher than a predetermined first threshold, and the representative value of the score of the plurality of second unlearned data is lower than the first threshold, n unlearned data are selected from the first group and m (m>n) unlearned data are selected from the second group. (Feature 3) By clustering the feature data of the multiple second untrained data sets, the multiple second untrained data sets are further divided into multiple subgroups. The processing apparatus according to feature 2, which selects one or more of the second unlearned data from each of the aforementioned subgroups. (Feature 4) The processing apparatus according to any one of features 1 to 3, which increases the number of unlearned data by expanding the selected unlearned data. (Feature 5) A processing device according to any one of features 1 to 4, which selects the unlearned data from each of the aforementioned multiple groups based on the score. (Feature 6) The user interface including the selected unlearned data is displayed on the display device. Through the user interface, the system accepts instructions for the selected unlearned data. An apparatus according to any one of features 1 to 5, which updates the first model using the selected unlearned data and the teaching. (Feature 7) The processing device described in any one of features 1 to 6, wherein each of the aforementioned multiple untrained data is image data. (Feature 8) A processing apparatus described in any one of features 1 to 7, The first model and the storage device storing the plurality of unlearned data, A display device that displays a user interface including the unlearned data selected by the processing device, An input device that receives instructions for the selected unlearned data, A processing system equipped with [the following features]. (Feature 9) In the processing unit, The trained first model is given multiple untrained data points, and the multiple output results produced by the first model are obtained. Using the aforementioned multiple output results, calculate multiple scores for the aforementioned multiple untrained data, Extract the feature data from each of the aforementioned multiple untrained data sets. By clustering multiple feature data, the multiple untrained data are divided into multiple groups. Depending on the score of the unlearned data included in each of the plurality of groups, one or more of the unlearned data are selected from each of the plurality of groups. Processing method. (Feature 10) A program that causes the processing unit to execute the processing method described in Feature 9. (Feature 11) A storage medium containing the program described in Feature 10.

[0084] According to the embodiments described above, a processing device, a processing system, a processing method, a program, and a storage medium are provided that can select unlearned data suitable for updating a model.

[0085] In this specification, "or" indicates that "at least one" of the items listed in the text may be adopted.

[0086] Although several embodiments of the present invention have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other. [Explanation of Symbols]

[0087] 1: Processing system, 10: Processing unit, 20: Input device, 30: Display device, 40: Storage device, 90: Computer, 100: Graphical user interface, 110: Selection area, 111: Pull-down icon, 112: Icon, 120: Selection area, 121: Icon, 123: Display area, 124: Icon, 125: Display area, 126: Icon, 130: Selection area, 131: Pull-down icon, 132: Icon, 200: Handling system, 210: Handling robot, 211: Robot arm, 211a: Link, 211b: Rotation axis, 212: Base, 213: Sensor, 215: Gripping part, 216: Suction mechanism, 216a: Suction pad, 217: Clamping mechanism, 217a: Support part, 218: Switching mechanism, 220: Camera, 230: Processing unit, C1, C2: Container, DF: Defect, FD: Feature data, IMG, IMG1, IMG2: Image, M1: First model, O: Object, O1: Object

Claims

1. Multiple untrained data points that were not used in training are input to the first trained model, and multiple output results output from the first model are obtained. Using the above-mentioned multiple output results, calculate the above-mentioned multiple scores for the above-mentioned untrained data, By clustering the feature data extracted from each of the aforementioned multiple untrained data sets, the aforementioned multiple untrained data sets are divided into multiple groups. Select one or more of the unlearned data from each of the aforementioned groups, A processing device in which, in the selection of unlearned data, the number of unlearned data selected from each of the plurality of groups is determined according to the score of the unlearned data included in each of the plurality of groups.

2. The aforementioned plurality of groups include the first group and the second group, The processing apparatus according to claim 1, wherein a plurality of first unlearned data included in the plurality of unlearned data are divided into a first group, a plurality of second unlearned data included in the plurality of unlearned data are divided into a second group, and when the representative value of the score of the plurality of first unlearned data is higher than a predetermined first threshold, and the representative value of the score of the plurality of second unlearned data is lower than the first threshold, n unlearned data are selected from the first group and m (m > n) unlearned data are selected from the second group.

3. By clustering the feature data of the multiple second untrained data sets, the multiple second untrained data sets are further divided into multiple subgroups. The processing apparatus according to claim 2, wherein one or more of the second unlearned data are selected from each of the plurality of subgroups.

4. The processing apparatus according to any one of claims 1 to 3, wherein the number of unlearned data is increased by expanding the selected unlearned data.

5. The processing apparatus according to any one of claims 1 to 3, wherein the unlearned data is selected from each of the plurality of groups based on the score.

6. The user interface including the selected unlearned data is displayed on the display device. Through the user interface, the system accepts instructions for the selected unlearned data. The processing apparatus according to any one of claims 1 to 3, which updates the first model using the selected unlearned data and the teaching.

7. The processing apparatus according to any one of claims 1 to 3, wherein each of the plurality of untrained data is image data.

8. The processing apparatus according to any one of claims 1 to 3, The first model and the storage device storing the plurality of unlearned data, A display device that displays a user interface including the unlearned data selected by the processing device, An input device that receives instructions for the selected unlearned data, A processing system equipped with [the following features].

9. In the processing unit, The trained first model is given multiple untrained data points, and the multiple output results output from the first model are obtained. Using the aforementioned multiple output results, calculate multiple scores for the aforementioned multiple untrained data, Extract the feature data from each of the aforementioned multiple untrained data sets. By clustering multiple feature data, the multiple untrained data are divided into multiple groups. Depending on the score of the unlearned data included in each of the aforementioned multiple groups, one or more of the unlearned data are selected from each of the aforementioned multiple groups. Processing method.

10. A program that causes a processing device to execute the processing method described in claim 9.

11. A storage medium storing the program described in claim 10.

Images