EVALUATION APPARATUS, SYSTEM, EVALUATION METHOD, AND PROGRAM

Abstract

To provide an evaluation device that can objectively evaluate results of simulation learning, and a system, a method for evaluation, and a program that use the evaluation device.SOLUTION: An evaluation device is for evaluating output of simulation learning to which action data is applied as teacher data, the action data having a structure in which a state vector showing a state and an action vector showing an action in that state are related to each other. The evaluation device has an evaluation unit for generating a numerical value that shows the evaluation of the output of the simulation learning on the basis of specific action data and the teacher data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an evaluation device, a system, an evaluation method, and a program. [Background technology]

[0002] Imitation learning is known as a technology for automating control that is difficult to express with a physical model. For example, Patent Document 1 discloses a technology for acquiring behavioral data that associates a state vector representing an environmental state with an action vector to be performed in that state, performing imitation learning, and outputting an action that is estimated to be performed in a certain state using the results of the imitation learning. Patent Document 2 discloses a technology for automatically winding power / communication cables in an aligned manner using imitation learning or the like. In imitation learning, it is desirable to use, for example, the behavior of a skilled operator as behavioral data as teacher data, and output estimated data that appropriately imitates this teacher data as a result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 150452 [Patent Document 2] International Publication No. 2020 / 027154 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is no established method for quantitatively confirming to what extent the inferred behavior data output as a result of imitation learning can imitate the operator. Therefore, the degree of imitation of the inferred data needs to be evaluated by someone who is familiar with the behavior to be performed in a certain situation, such as a skilled operator, and there is a risk that the objectivity of the evaluation is low. In addition, when the evaluation of the degree of imitation has not been established, inferred behavior data with a low degree of imitation may be mistakenly used for control, and safety problems may occur due to incorrect control.

[0005] The present invention has been made in consideration of the above, and has an object to provide an evaluation device that can objectively evaluate the results of imitation learning, a system that uses the same, and an evaluation method and program. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, one aspect of the present invention is an evaluation device that evaluates the output of imitation learning by applying behavioral data as teacher data, the behavioral data having a structure that corresponds a state vector representing a state with a behavior vector representing an action in the state, and the evaluation device is equipped with an evaluation unit that generates a numerical value representing the evaluation of the output of the imitation learning as evaluation information based on specified behavioral data and the teacher data.

[0007] The evaluation unit may use inferred behavior data included in an output of the imitation learning as the predetermined behavior data, and generate an evaluation value indicating a level of proficiency in the imitation learning as the evaluation information.

[0008] The evaluation unit may generate the evaluation value based on a distance between an element of a state vector included in the teacher data and an element of an inferred state vector included in the inferred behavior data.

[0009] The evaluation section may include a model generation section that generates an evaluation model for generating the evaluation information.

[0010] The model generation unit may perform data processing using similarity as an index for pairs of elements of state vectors included in each of the plurality of behavioral data and elements of behavior vectors corresponding to the elements, to generate new behavioral data.

[0011] The similarity may be measured using a distance between elements of state vectors included in each of the plurality of behavioral data as an index, and elements may be removed from the new behavioral data based on the distance.

[0012] The model generation section may generate the evaluation model by using evaluation data in which behavioral data and evaluation values ​​for the behavioral data correspond to each other based on a mathematical model.

[0013] The evaluation model may be generated by supervised machine learning.

[0014] One aspect of the present invention is a system including the evaluation device and a learning device that has an imitation learning model and generates an output of the imitation learning.

[0015] The learning device may constitute a control device for a winding device for flexible material.

[0016] One aspect of the present invention is an evaluation method for evaluating the output of imitation learning that applies behavioral data as teacher data having a structure that corresponds a state vector representing a state with a behavior vector representing an action in the state, and the evaluation method includes an evaluation step of generating a numerical value representing an evaluation of the output of the imitation learning as evaluation information based on specified behavioral data and the teacher data.

[0017] The evaluation step may use inferred behavior data included in an output of the imitation learning as the predetermined behavior data, and generate an evaluation value indicating a level of proficiency in the imitation learning as the evaluation information.

[0018] The evaluation step may include a model generation step of generating an evaluation model for generating the evaluation information.

[0019] One aspect of the present invention is a program that executes a step of evaluating the output of imitation learning that applies behavioral data as teacher data, the behavioral data having a structure that corresponds a state vector representing a state with a behavior vector representing an action in the state, the step including a step of generating a numerical value representing an evaluation of the output of the imitation learning as evaluation information based on specified behavioral data and the teacher data. Effect of the Invention

[0020] According to the present invention, it is possible to objectively evaluate the results of imitation learning. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing the configuration of a system including an evaluation device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of an information processing device that can constitute each of the evaluation device and the learning device. [Diagram 3] FIG. 3 is a diagram illustrating an example of the behavior data and the inferred behavior data. [Figure 4] FIG. 4 is a flow diagram illustrating an example of a process executed by the evaluation device. [Diagram 5] FIG. 5 is a flow diagram showing an example of the evaluation information generation process in FIG. [Figure 6] FIG. 6 is an explanatory diagram showing an example of the selected data. [Figure 7] FIG. 7 is a diagram showing the configuration of a system including an evaluation device according to the second embodiment. [Figure 8] FIG. 8 is a flow diagram showing an example of control executed by the evaluation device. [Figure 9] FIG. 9 is an explanatory diagram showing an example of generation of low skill level data. [Figure 10] FIG. 10 is an explanatory diagram of the improvement of the quality of low-proficiency data. [Figure 11] FIG. 11 is a flow diagram showing an example of a process for improving the quality of low proficiency data. [Figure 12] FIG. 12 is a diagram illustrating an example of the evaluation data. [Figure 13] FIG. 13 is a diagram showing a configuration of a system according to the third embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of the operation of the system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are appropriately assigned the same reference numerals. Moreover, the drawings are schematic, and the dimensional relationship of each element, the ratio of each element, and the like may differ from reality. Furthermore, between the drawings, there may be parts whose dimensional relationship and ratio differ from each other.

[0023] (Embodiment 1) 1 is a diagram showing the configuration of a system including an evaluation device according to embodiment 1. The evaluation device 10 according to embodiment 1 constitutes a system 100 together with a learning device 20. The learning device 20 performs imitation learning and outputs the results thereof, and also generates an imitation-trained model. The evaluation device 10 evaluates the output of the imitation learning of the learning device 20.

[0024] [Device configuration] 2 is a diagram showing an example of the configuration of an information processing device that can constitute each of the evaluation device 10 and the learning device 20. The information processing device 30 includes a processor 31, a memory 32, and an interface 33.

[0025] The processor 31 includes, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), and a graphics processing unit (GPU), and performs various arithmetic processing for control. The memory 32 includes a storage unit such as a ROM in which various programs and data used by the processor 31 to perform arithmetic processing are stored, and a storage unit such as a RAM used for storing a working space when the processor 31 performs arithmetic processing and results of the arithmetic processing of the processor 31. The memory 32 may include an auxiliary storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0026] The interface 33 connects the information processing device 30 to an input / output device and a communication network. The interface 33 includes an interface board for connecting to an input device such as a mouse and an output device such as a display, a communication board for performing information communication by wire or wirelessly, etc. The communication network is, for example, a LAN (Local Area Network).

[0027] 1, the evaluation device 10 includes, as its functional units, an acquisition unit 11, an evaluation unit 12, and an output unit 13. The learning device 20 includes, as its functional units, an acquisition unit 21, a learning unit 22, and an output unit 23. These functional units may be realized, for example, by a processor 31 of the information processing device 30 executing a program, whereby hardware and software work together, or may be realized as functions of the hardware of the information processing device 30.

[0028] [Learning device functions] First, there will be described the functions of the functional units of the learning device 20. In the learning device 20, the acquisition unit 21 acquires, via the interface 33, for example, behavioral data to be used for imitation learning.

[0029] The behavior data will be described with reference to FIG. 3(a). The behavior data has a structure in which a state vector representing a state corresponds to an action vector representing an action in that state. In the example of FIG. 3(a), the behavior data A is in a table data format. In the behavior data A, Asc (state current) is a state vector, and its components are state elements that are parameters representing a state. Aac (action current) is an action vector, and its components are action elements that are parameters representing an action. The state vector is represented by (Asc1, Asc2, . . .), and the action vector is represented by (Aac1, Aac2, . . .). And the action element representing the action in the state represented by the state element Asc1 is Aac1.

[0030] Furthermore, state element Asc2 represents a state transitioned to by executing state element Asc1. For example, if state element Asc1 represents the initial position of the control object and action element Aac1 represents the distance to move the control object, then for Asn(state next)1 described below, Asn1=Asc2=Asc1+Aac1 holds. Also, if m and k are integers of 2 or more, and the state vector and action vector are m-dimensional, then Asn(k-1)=Asck=Asc(k-1)+Aac(k-1) holds for k less than m.

[0031] In the following, for example, Asc(k-1) may be described as a current state element, Asck as Asn(k-1), and also as a next state element.

[0032] The acquisition unit 21 acquires, for example, behavioral data A as behavioral data to be used in imitation learning. The behavioral data A is, for example, data indicating a control behavior when a skilled operator performs good aligned winding when manually controlling a cable aligned winding device.

[0033] The learning unit 22 performs imitation learning by applying the behavior data acquired by the acquisition unit 21 as teacher data, generates an imitation-learned model, and generates behavior data predicted by applying the imitation-learned model (predicted behavior data). When generating the predicted behavior data, the state elements as initial values ​​may be input from the outside via the interface 33, or may be read from the memory 32.

[0034] The output unit 23 outputs the inferred behavior data generated by the learning unit 22, and displays it or transmits it to the outside via the interface 33, for example.

[0035] The inferred behavior data will be described with reference to FIG. 3(b). In the example of FIG. 3(a), in the inferred behavior data B, Bsc is an inferred state vector having inferred inferred state elements as components. However, only Bsc1 is given as an initial value, but for convenience, it is defined as an inferred state element. Note that Asc1 may be given as Bsc1. Bac is an inferred behavior vector, and has inferred state elements, which are parameters representing inferred behavior, as components. Then, in the state represented by the inferred state element Bsc1, the inferred behavior element representing the behavior inferred as a result of imitation learning is Bac1. Then, the inferred state element Bsc2 represents the state inferred to transition to by executing the inferred state element Bsc1. The inferred state vector is represented by (Bsc1, Bsc2, . . . ), and the inferred behavior vector is represented by (Bac1, Bac2, . . . ).

[0036] The inferred state element Bsc1 and the inferred action element Bac1 are associated with each other by, for example, a policy function. As in the case of the behavior data A, when the inferred state element Bsc1 represents the initial position of the control object and the inferred action element Bac1 represents the estimated value of the distance to move the control object, for Bsn1 described later, Bsn1=Bsc2=Bsc1+Bac1 holds. In addition, if m and k are integers and the inferred state vector and the inferred action vector are m-dimensional, Bsn(k-1)=Bsck=Bsc(k-1)+Bac(k-1) holds for k less than m.

[0037] In the following, for example, Bsc(k-1) may be referred to as the current guess state element, Bsck as Bsn(k-1), and also as the next guess state element.

[0038] [Functions of evaluation device] Next, a description will be given of the functions of each functional unit of the evaluation device 10. In the evaluation device 10, the acquisition unit 11 acquires, for example, via the interface 33, behavioral data A, which is behavioral data used as teacher data in the learning device 20, and acquires the output of imitation learning including inferred behavioral data B. The inferred behavioral data B is an example of predetermined behavioral data.

[0039] Based on the behavior data A and the inferred behavior data B, the evaluation unit 12 generates a numerical value representing an evaluation of the output of the imitation learning of the learning device 20 as evaluation information.

[0040] The output unit 13 outputs the evaluation information generated by the evaluation unit 12, and displays or transmits it to the outside via, for example, an interface 33. The evaluation information may be stored in the memory 32.

[0041] [Processing flow] 4 is a flow diagram showing an example of a process executed by the evaluation device 10. First, in step S101, the acquisition unit 11 acquires teacher data and an output of imitation learning. Then, in step S102, the evaluation unit 12 generates evaluation information (an example of an evaluation step). Then, in step S103, the output unit 13 outputs the evaluation information.

[0042] A specific example will be given of the generation of evaluation information performed by the evaluation unit 12 in step S102. As an example, the evaluation unit 12 generates an evaluation value indicating a proficiency level of imitation learning as the evaluation information. The evaluation value is generated based on, for example, a distance between an element of a state vector included in behavior data A as teacher data and an element of an inferred state vector included in inferred behavior data B.

[0043] Fig. 5 is a flow diagram showing an example of the generation process of the evaluation information in Fig. 4. First, in step S201, the evaluation unit 12 selects a pair of state elements (Bscs, Bsns) from the inferred behavior data B. Here, s indicates the selected element.

[0044] Next, in step S202, the evaluation unit 12 selects a designated number of data (Ascs) from the state elements of the teacher data in order of smallest Euclidean distance from Bscs. Here, the designated number is not particularly limited as long as it is plural, but is, for example, 3.

[0045] Next, in step S203, the evaluation unit 12 calculates the Euclidean distance between Asns and Bsns for all the selected data (Ascs).

[0046] Next, in step S204, the evaluation unit 12 judges whether the calculation of the Euclidean distance has been performed for all selected (Bscs, Bsns). If it is judged that the calculation has not been performed (step S204, No), the evaluation unit 12 selects (Bscs, Bsns) that has not yet been selected in step S205, and the process returns to step S202. If it is judged that the calculation has been performed (step S204, Yes), the process proceeds to step S206.

[0047] Next, in step S206, the evaluation unit 12 generates an evaluation value indicating the degree of proficiency. In this example, the average of all the calculated Euclidean distances is set as the evaluation value of the degree of proficiency. However, the evaluation value is not limited to this, and for example, the minimax method may be applied to all the calculated Euclidean distances, and the Euclidean distance to the farthest coordinate may be set as the evaluation value. After that, the evaluation unit 12 ends the processing flow. The degree of proficiency may be normalized so that it takes values ​​between 0 and 1 continuously.

[0048] Here, the processing of the selected data will be described with reference to Fig. 6. Fig. 6(a) shows a case where a pair of state elements (Bsc1, Bsn1) is selected from the inferred behavior data B, and three pieces of data (Asc1, Asc2, Asc3), which is an example of a specified number, are selected in order of the smallest Euclidean distance from Bsc1. In this case, the evaluation unit 12 calculates the Euclidean distance between Asns and Bsns for the pieces of data (Asn1, Asn2, Asn3).

[0049] In the example shown in Figure 6(a), when a current state element (Asc1, Asc2, Asc3) with a small Euclidean distance to the current guessed state element (Bsc1) is selected, the Euclidean distance of the corresponding next state element (Asn1, Asn2, Asn3) to the next guessed state element (Bsn1) is also small, so it can be determined that the imitation learning was successful and that guessed data with a relatively high level of proficiency was obtained as the output of the imitation learning.

[0050] On the other hand, the example shown in Figure 6(b) is a case where a current state element (Asc1, Asc2, Asc3) with a small Euclidean distance to a current guess state element (Bsc1) is selected, but the Euclidean distance of the next state element (Asn1, Asn2, Asn3) to the corresponding next guess state element (Bsn1) is large. In this case, it can be determined that guess data has been obtained as the output of imitation learning with a relatively low level of proficiency.

[0051] As described above, the evaluation device 10 has an evaluation unit 12 that generates evaluation information representing an evaluation of the output of the imitation learning of the learning device 20 used to generate the inferred behavior data B based on the behavioral data A and the inferred behavior data B, as an evaluation value indicating the level of proficiency in imitation learning, thereby making it possible to objectively evaluate the level of proficiency in imitation learning.

[0052] In particular, the evaluation unit 12 generates an evaluation value based on the Euclidean distance between the state elements (current state element and next state element) included in the behavioral data A and the inferred state elements (current inferred state element and next inferred state element) included in the inferred behavior data B, and is therefore able to provide an objective evaluation based on a quantitative evaluation value.

[0053] In the above embodiment, the evaluation value is generated using the Euclidean distance, but the evaluation value may be generated using a distance other than the Euclidean distance, for example, a Manhattan distance, a Chebyshev distance, a Mahalanobis distance, etc. In other words, the distance may be appropriately selected and used from a distance function defined by a non-negative real value function.

[0054] (Embodiment 2) 7 is a diagram showing the configuration of a system including an evaluation device according to embodiment 2. An evaluation device 40 according to embodiment 2 constitutes a system 200 together with a learning device 20. The learning device 20 performs imitation learning, and an evaluation device 10 evaluates the output of the imitation learning.

[0055] [Device configuration] Each of the evaluation device 40 and the learning device 20 can be configured by an information processing device 30, similar to the case of the system 100 of FIG.

[0056] Since the learning device 20 is the same as the learning device 20 in FIG. 1, detailed description will be omitted. The evaluation device 40 includes, as functional units, an acquisition unit 41, an evaluation unit 42, and an output unit 43. The evaluation unit 42 includes a model generation unit 42a. The model generation unit 42a includes a data generation unit 42a1, an evaluation / processing unit 42a2, and a learning unit 42a3. The data generation unit 42a1 includes a learning unit 42a11. These functional units may be realized, for example, by the processor 31 of the information processing device 30 executing a program, whereby hardware and software work together, or may be realized as functions of the hardware of the information processing device 30.

[0057] [Functions of evaluation device] Next, we will explain the function of each functional unit of evaluation device 40. In evaluation device 40, acquisition unit 41, like acquisition unit 11 of evaluation device 10, acquires behavioral data, such as behavioral data A, to be used as teacher data in learning device 20, via interface 33, for example.

[0058] In the model generation unit 42a of the evaluation unit 42, the data generation unit 42a1 generates inferred behavior data by having a learning unit 42a11 equipped with an imitation learning model perform imitation learning using the behavior data A as teacher data. The imitation learning model equipped in the learning unit 42a11 may be the same as the imitation learning model equipped in the learning unit 22 of the learning device 20, but is not limited thereto.

[0059] However, the inferred behavior data generated at this time has a low degree of imitation, i.e., a low level of proficiency. Hereinafter, inferred behavior data with a low level of proficiency may be referred to as low-proficiency data C. The data structure and generation method of the low-proficiency data C will be described in detail later.

[0060] The evaluation / processing unit 42a2 processes data such as low-proficiency data generated by the data generating unit 42a1 to generate new behavioral data, or generates evaluation data to be used by the learning unit 42a3. The evaluation / processing unit 42a2 can also evaluate the output of the imitation learning using the evaluation model generated by the learning unit 42a3, and generate a numerical value representing the evaluation of the output of the imitation learning as evaluation information. A method for generating new behavioral data and evaluation data, and a method for evaluating the output of the imitation learning will be described in detail later. The evaluation / processing unit 42a2 is an example of an evaluation data generating unit that generates evaluation data. The evaluation / processing unit 42a2 is also an example of an evaluation unit.

[0061] The learning unit 42a3 includes a learning model for performing supervised machine learning. The learning unit 42a3 uses the evaluation data generated by the evaluation / processing unit 42a2 as teacher data to perform supervised machine learning to generate an evaluation model.

[0062] The output unit 43 outputs the evaluation information (such as the evaluation value) generated by the evaluation / processing unit 42a2, and displays or transmits it to the outside via the interface 33, for example.

[0063] [Processing flow] 8 is a flow diagram showing an example of processing executed by the evaluation device. This processing flow includes step S301 of acquiring teacher data, step S302 of generating low proficiency data, step S303 of improving the quality of the low proficiency data, step S304 of generating evaluation data, step S305 of generating an evaluation model, and step S306 of generating and outputting an evaluation value.

[0064] A specific description will be given below. First, in step S301, the acquisition unit 41 acquires teacher data. Then, in step S302, the data generation unit 421 generates low proficiency data C.

[0065] 9 is an explanatory diagram showing an example of low proficiency data generation. For example, the data generation unit 42a1 uses the behavior data A as teacher data and causes the learning unit 42a11 equipped with an imitation learning model to perform imitation learning to generate low proficiency data C, which is inferential behavior data with a low proficiency. The imitation learning model equipped in the learning unit 42a11 may be the same as the imitation learning model equipped in the learning unit 22 of the learning device 20, but is not limited to this.

[0066] The low proficiency data C can be generated under conditions considered to be low proficiency, for example, by stopping imitation learning at a relatively early stage, i.e., when imitation learning has not progressed much. In this case, for example, if the state vectors and action vectors constituting the behavior data A are m-dimensional, the inferred behavior data with low proficiency can be expressed as behavior data with state vectors and action vectors of l-dimension, where l is an integer smaller than m.

[0067] In the low proficiency data C illustrated in FIG. 9, Csc is an inferred state vector having components of inferred inferred state elements. However, only Csc1 is given as an initial value, but for convenience, it is defined as an inferred state element. Note that Asc1 may be given as Csc1. Cac is an inferred action vector, and has components of inferred state elements, which are parameters representing inferred actions. Then, in the state represented by the inferred state element Csc1, an inferred action element representing an action inferred using an imitation learning model or an initial stage imitation learned model is Cac1. Then, an inferred state element Csc2 represents a state inferred to transition by executing the inferred state element Csc1. The inferred state vector is represented by (Csc1, Csc2, Csc3...), and the inferred action vector is represented by (Cac1, Cac2, Cac3...).

[0068] The estimated state element Csc1 and the estimated action element Cac1 are associated with each other by, for example, a policy function. As in the case of the behavior data A, when the estimated state element Csc1 represents the initial position of the control object and the estimated action element Cac1 represents the estimated value of the distance to move the control object, Csn1=Csc2=Csc1+Cac1 holds for Csn1 described later. In addition, if l and k are integers and the estimated state vector and the estimated action vector are l-dimensional, Csn(k-1)=Csck=Csc(k-1)+Cac(k-1) holds for k less than l.

[0069] In the following, for example, Csc(k-1) may be referred to as the current guess state element, Csck as Csn(k-1), and also as the next guess state element.

[0070] Returning to FIG. 8, in step S303, the evaluation / processing unit 42a2 improves the low skill level data C generated by the data generating unit 42a1 to generate low skill level data C' as new behavioral data.

[0071] The low proficiency data C is data generated under conditions that are considered to be low proficiency. However, the combinations of state elements and behavior elements contained in the low proficiency data C may unintentionally include combinations that are the same as or similar to combinations of state elements and behavior elements contained in behavior data with high proficiency, for example, in teacher data. In such cases, the low proficiency data C may not be considered good data with low proficiency.

[0072] FIG. 10 is an explanatory diagram of the improvement of the quality of low proficiency data C. For example, assume that the state vector Csc represents the X coordinate of the position of the control object, and the action vector Cac represents the distance to move the control object. In this case, as shown in FIG. 10(a), if the next state element of the low proficiency data C is the X coordinate after the movement, the X coordinate is distributed evenly in a relatively wide range from about +1 to +5, and the distribution of the data of the state element is not so different for each X coordinate. Here, the data ratio on the vertical axis is the ratio of the number of state elements that are the value of the X axis to the total number of state elements in the target action data. Such a distribution is due to the low proficiency of the low proficiency data C.

[0073] On the other hand, in highly skilled behavioral data that can be used as teacher data such as behavioral data A, most of the state element data will have the same value at the X coordinate after the movement, which is +3 in this example. In this case, data in low skilled data C whose X coordinate is +3 or close to it will be the same as the highly skilled behavioral data, which is not preferable.

[0074] Therefore, in order to improve the quality of the low skill level data C, data processing is performed to remove data in the low skill level data C whose X coordinate is +3 or in the vicinity thereof, as shown in FIG. 10(b).

[0075] FIG. 11 is a flow diagram showing an example of a quality improvement process for the low skill level data C in FIG.

[0076] First, in step S401, the evaluation / processing unit 42a2 selects a pair of state elements (Cscs, Csns) from the low skill data C, which is inferred behavior data. Here, s indicates the selected element.

[0077] Next, in step S402, the evaluation / processing unit 42a2 selects a designated number of data (Ascs) from the state elements of the teacher data in order of the smallest Euclidean distance from Cscs. The designated number is not particularly limited as long as it is plural, but is, for example, 3.

[0078] Next, in step S403, the evaluation / processing unit 42a2 calculates the Euclidean distance between Asns and Csns for all the selected data (Ascs).

[0079] Next, in step S404, the evaluation / processing unit 42a2 judges whether the Euclidean distance calculation has been performed for all selected (Cscs, Csns). If it is judged that the calculation has not been performed (step S404, No), in step S405, the evaluation / processing unit 42a2 selects (Cscs, Csns) that has not yet been selected, and the process returns to step S402. If it is judged that the calculation has been performed (step S404, Yes), the process proceeds to step S406.

[0080] Next, in step S406, the evaluation / processing unit 42a2 selects and excludes data with high proficiency (Cscs, Csns). In this example, the average of the calculated Euclidean distances is set as the proficiency as the evaluation value. However, the evaluation value is not limited to this, and for example, the minimax method may be applied to all the calculated Euclidean distances, and the Euclidean distance to the farthest coordinate may be set as the evaluation value. Then, data with high proficiency (Cscs, Csns) with a proficiency higher than a predetermined threshold (average Euclidean distance is smaller than a predetermined threshold) is selected and excluded. In this way, new low proficiency data C' is generated from which data with high proficiency has been excluded. After that, the evaluation / processing unit 42a2 ends the processing flow.

[0081] 11, data processing is performed using similarity as an index for pairs of elements (state elements) of state vectors included in each of the multiple behavioral data (behavioral data A and low proficiency data C) and elements (behavior elements) of behavioral vectors corresponding to the elements, to generate new behavioral data (low proficiency data C'). In the quality improvement processing of FIG 11, similarity is measured using the distance (Euclidean distance) between elements of state vectors included in each of the multiple behavioral data (behavioral data A and low proficiency data C), and elements (elements with high proficiency) are removed based on the distance in the new behavioral data (low proficiency data C').

[0082] The description will be continued by returning to Fig. 8. In step S304, the evaluation / processing unit 42a2 generates evaluation data.

[0083] 12 is a diagram showing an example of evaluation data. This evaluation data uses behavioral data A and low proficiency data C' as a plurality of behavioral data. Then, for a pair of a state element included in the behavioral data A and the low proficiency data C' and a corresponding behavior element, a proficiency level is associated as an evaluation value.

[0084] Specifically, all of the combinations of state elements and behavior elements included in the behavior data A have a high level of proficiency. Therefore, they are labeled and associated with "1", which is the highest standardized proficiency value. Also, all of the combinations of state elements and behavior elements included in the low proficiency data C' have a low level of proficiency. Therefore, they are labeled and associated with "0", which is the lowest standardized proficiency value, as an evaluation index. Such association may be performed, for example, using a mathematical model used in machine learning.

[0085] The evaluation data shown in FIG. 12 has a data structure in which a pair of a state element and a behavior element is an explanatory variable, and a label (proficiency) is an objective variable.

[0086] In addition, in the evaluation data shown in FIG. 12, there are two types of labels, "0" and "1", but the evaluation data may be generated by further combining other behavioral data whose proficiency levels take values ​​other than "0" and "1".

[0087] Returning to Fig. 8, the description will be continued. In step S305, the learning unit 42a3 performs supervised machine learning using the evaluation data as shown in Fig. 12 as teacher data to generate an evaluation model (an example of a model generation step). The algorithm of the supervised machine learning is not particularly limited, and for example, a known mathematical model such as a neural network, a kNN method, a linear model, a Naive Bayes classifier, a decision tree, or an SVM (support vector machine) can be used. That is, the evaluation model associates the state vector and the action vector included in the action data with the evaluation value for the action vector based on the mathematical model.

[0088] The evaluation model generated by performing supervised machine learning using the evaluation data as training data as shown in Fig. 12 can evaluate behavioral data such as inferred behavior data in terms of proficiency. The proficiency can take a continuous value between 0 and 1.

[0089] Next, in step S306, the evaluation / processing unit 42a2 accepts input of inferential behavior data (e.g., inferential behavior data B) which is the output of the imitation learning by the learning device 20 acquired by the output unit 43, and generates a proficiency level which is evaluation information for the inferential behavior data B as an evaluation value. The output unit 43 outputs the proficiency level generated by the evaluation / processing unit 42a2, and displays it via the interface 33 or transmits it to the outside, for example. The evaluation information may be stored in the memory 32.

[0090] As described above, the evaluation device 40 has an evaluation / processing unit 42a2 that generates evaluation information representing an evaluation of the output of the imitation learning of the learning device 20 used to generate the inferred behavior data B based on the behavioral data A and the low proficiency data C as an evaluation value indicating the proficiency in imitation learning, thereby making it possible to objectively evaluate the proficiency in imitation learning.

[0091] In particular, since the evaluation unit 12 generates an evaluation value using the improved low skill level data C', it is possible to provide an objective evaluation based on a quantitative evaluation value.

[0092] In the above embodiment, the quality is improved using the Euclidean distance, but distances other than the Euclidean distance, such as the Manhattan distance, the Chebyshev distance, or the Mahalanobis distance, may also be used.

[0093] (Embodiment 3) 13 is a diagram showing the configuration of a system according to embodiment 3. The system 1000 includes a winding device 300 and a control device 400 that controls the winding device 300.

[0094] The winding device 300 is a device that winds a power cable C, which is an example of a flexible object, around a drum D. The drum D has a cylindrical body D1 and two flanges D2 provided on both ends of the body D1. The power cable C is wound around the body D1.

[0095] The winding device 300 includes a rotation mechanism 310 that rotates the drum D about its rotation axis by a rotation actuator, an arm 320 that guides the power cable C to the drum D, a guide rail 330 that guides the arm 320 along the rotation axis direction of the drum D, and a movement mechanism 340 that moves the arm 320 along the guide rail 330 by a rotation actuator. In Fig. 13, the rotation axis direction of the drum D and the extension direction of the guide rail 330 are defined as the X direction.

[0096] Arm 320 guides power cable C to body portion D1 by reciprocating on guide rail 330 in the positive and negative directions of the X direction, and also in the positive and negative directions of the Y and Z directions perpendicular to the X direction.

[0097] The control device 400 controls the position and movement amount of the arm 320 by controlling the rotation actuator of the movement mechanism 340. The control device 400 controls the rotation actuator in a substantially automatic manner, but is configured to allow an operator to manually intervene in the control in order to make the winding state of the power cable C a suitable state.

[0098] Specifically, the control device 400 includes a manual operation unit 410 , a manual control unit 420 , an automatic control unit 430 , an evaluation unit 440 , and a notification unit 450 .

[0099] The manual operation unit 410 includes operators such as buttons and levers, and is a section where an operator performs manual operations to manually intervene in the control.

[0100] The manual control unit 420, the automatic control unit 430, and the evaluation unit 440 are configured using, for example, an information processing device 30 as shown in FIG. 2. The manual control unit 420 receives a manual operation signal from the manual operation unit 410, and controls the rotation actuator of the moving mechanism 340 in response to the operation signal. The automatic control unit 430 is provided with a trained model 431 generated by imitation learning using behavioral data as teacher data, and automatically controls the rotation actuator of the moving mechanism 340 based on the estimation result by the trained model 431. The manual control unit 420 and the automatic control unit 430 work in cooperation with each other so that the manual control can smoothly intervene in the automatic control. The automatic control unit 430 outputs the estimation result by the trained model 431 as behavioral data. The automatic control unit 430 is an example of a learning device.

[0101] The evaluation unit 440 is an example of an evaluation device, and receives behavioral data of the inference result from the automatic control unit 430, and generates a proficiency level for the behavioral data as an evaluation value. This behavioral data may be updated in real time and provided by the automatic control unit 430, or may be updated by the evaluation unit 440. The notification unit 450 includes, for example, a display, a lamp, and a speaker, and issues a notification to encourage an operator to intervene.

[0102] When the generated proficiency level remains below a predetermined value for a certain period of time, the evaluation unit 440 controls the notification unit 450 to notify the operator to intervene in the control. As a result, the operator can intervene in the control as necessary, and the winding state of the power cable C can be made into an optimal state.

[0103] Fig. 14 is an explanatory diagram of an example of the operation of the system 1000 shown in Fig. 13. The horizontal axis of Fig. 14 indicates the elapsed time from the start of winding control, and the vertical axis indicates the intervention action and the proficiency level. In the system 1000, when the proficiency level falls below a predetermined value for a certain period of time, a notification is issued to encourage manual intervention, and when the intervention actions A and B are performed, the proficiency level can be restored.

[0104] An example of standardization of state elements and behavior elements in the system 1000 will now be described. The arm 320 moves in the XYZ directions. In this case, the state element Asc is defined by the x, y, and z coordinates of the arm 320, such as Asc: (x coordinate (0-1), y coordinate (0-1), z coordinate (0-1). At this time, the raw data values ​​of the x, y, and z coordinates are standardized so that the range of motion of the arm 320 is within the range of 0-1, so that the x, y, and z coordinates can take a range of 0-1. On the other hand, the behavior element Aac is also defined as Aac: (x coordinate (0-1), y coordinate (0-1), z coordinate (0-1). At this time, the raw data values ​​of the x, y, and z coordinates are standardized by dividing them by the maximum value of each coordinate of the behavior element Aac, so that the x, y, and z coordinates can take a range of 0-1.

[0105] The evaluation unit 440 may control the notification unit 450 to visually display the proficiency level or to notify the proficiency level by voice, thereby enabling the operator to confirm the proficiency level.

[0106] Although the winding device 300 of the system 1000 is a winding device for the power cable C, the configuration of the system 1000 can also be applied to winding devices for other flexible materials such as metal cables, fiber cables, strings, resin films, paper, and metal foils.

[0107] Furthermore, the present invention is not limited to the above-mentioned embodiment. The present invention also includes a configuration in which the above-mentioned components are appropriately combined. Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the above-mentioned embodiment, and various modifications are possible. [Explanation of symbols]

[0108] 10, 40: Evaluation device 11, 21, 41: Acquisition part 12, 42, 440: Evaluation section 13, 23, 43: Output section 20: Learning device 22, 42a11, 42a3: Learning Department 30: Information processing device 31: Processor 32: Memory 33: Interface 42a: Model generation section 42a1: Data generation section 42a2: Evaluation and processing section 100, 200, 1000: System 300: Winding device 310: Rotation mechanism 320: Arm 330: Guide rail 340: Movement mechanism 400: Control device 410: Manual operation section 420: Manual control section 430: Automatic control unit 431: Trained model 450: Information Department A: Intervention action B: Intervention action C: Power cable D: Drums D1: Torso D2:Tsubabe

Claims

1. 1. An evaluation device for evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state and a behavior vector representing an action in the state are associated with each other as teacher data, the evaluation device comprising: an evaluation unit that generates, as evaluation information, a numerical value representing an evaluation of an output of the imitation learning based on predetermined behavioral data and the teacher data; the evaluation unit uses inferred behavior data included in the output of the imitation learning as the predetermined behavior data, and generates an evaluation value indicating a proficiency of the imitation learning as the evaluation information; The evaluation unit generates the evaluation value based on a distance between an element of a state vector included in the teacher data and an element of an inferred state vector included in the inferred behavior data, the distance function being a non-negative real-valued function. Evaluation equipment.

2. An evaluation device for evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state corresponds to a behavior vector representing an action in the state, as teacher data, comprising: an evaluation unit that generates, as evaluation information, a numerical value representing an evaluation of an output of the imitation learning based on predetermined behavioral data and the teacher data; the evaluation unit includes a model generation unit that generates an evaluation model for generating the evaluation information, the model generation unit performs data processing using similarity as an index for pairs of elements of state vectors included in each of the plurality of pieces of behavior data and elements of behavior vectors corresponding to the elements of the state vectors, thereby generating new behavior data; The similarity is determined by using a distance between elements of state vectors included in each of the plurality of behavioral data as an index, and elements are removed from the new behavioral data based on the distance. Evaluation equipment.

3. The model generation unit generates evaluation data in which behavior data and an evaluation value for the behavior data correspond to each other based on a mathematical model, and generates the evaluation model using the evaluation data. The evaluation device according to claim 2 .

4. The evaluation model is generated by supervised machine learning. The evaluation device according to claim 3 .

5. An evaluation device according to any one of claims 1 to 3, A learning device that includes an imitation learning model and generates an output of the imitation learning; A system equipped with.

6. The learning device constitutes a control device for a winding device for flexible material. The system of claim 5.

7. An evaluation method for evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state corresponds to a behavior vector representing an action in the state, the evaluation method comprising the steps of: an evaluation step of generating a numerical value representing an evaluation of an output of the imitation learning as evaluation information based on predetermined behavioral data and the teacher data; In the evaluation step, inferred behavior data included in the output of the imitation learning is used as the predetermined behavior data, and an evaluation value indicating a proficiency level of the imitation learning is generated as the evaluation information; In generating the evaluation value, the evaluation value is generated based on a distance defined by a distance function that is a non-negative real-valued function between an element of a state vector included in the teacher data and an element of an inferred state vector included in the inferred behavior data. Evaluation method.

8. A method for evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state corresponds to a behavior vector representing an action in the state, the method comprising the steps of: an evaluation step of generating a numerical value representing an evaluation of an output of the imitation learning as evaluation information based on predetermined behavioral data and the teacher data; The evaluation step includes a model generation step of generating an evaluation model for generating the evaluation information, In the model generation step, data processing is performed using similarity as an index for pairs of elements of state vectors included in each of the plurality of pieces of behavior data and elements of behavior vectors corresponding to the elements of the state vectors to generate new behavior data; The similarity is determined by using a distance between elements of state vectors included in each of the plurality of behavioral data as an index, and elements are removed from the new behavioral data based on the distance. Evaluation method.

9. A program for executing a step of evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state and a behavior vector representing an action in the state are associated with each other as teacher data, the program comprising: The step includes an evaluation step of generating, as evaluation information, a numerical value representing an evaluation of an output of the imitation learning based on predetermined behavioral data and the teacher data, In the evaluation step, inferred behavior data included in the output of the imitation learning is used as the predetermined behavior data, and an evaluation value indicating a proficiency level of the imitation learning is generated as the evaluation information; In generating the evaluation value, the evaluation value is generated based on a distance defined by a distance function that is a non-negative real-valued function between an element of a state vector included in the teacher data and an element of an inferred state vector included in the inferred behavior data. program.

10. A program for executing a step of evaluating an output of imitation learning using behavioral data having a structure in which a state vector representing a state corresponds to a behavior vector representing an action in the state as teacher data, the program comprising: The step includes an evaluation step of generating, as evaluation information, a numerical value representing an evaluation of an output of the imitation learning based on predetermined behavioral data and the teacher data, The evaluation step includes a model generation step of generating an evaluation model for generating the evaluation information, In the model generation step, data processing is performed using similarity as an index for pairs of elements of state vectors included in each of the plurality of pieces of behavior data and elements of behavior vectors corresponding to the elements of the state vectors to generate new behavior data; The similarity is determined by using a distance between elements of state vectors included in each of the plurality of behavioral data as an index, and elements are removed from the new behavioral data based on the distance. program.

Images